Attachment 1458773 Details for Bug 1600138 – mon logs

mon logs

ceph-mon.magna102.log-20180712 (text/plain), 4.83 MB, created by Parikshith on 2018-07-13 18:03:07 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Parikshith

Created: 2018-07-13 18:03:07 UTC

Size: 4.83 MB

patch

obsolete

>2018-07-11 03:34:03.787476 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213438 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:04.059127 7f29149e4700  1 mon.magna102@0(leader).log v18040 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18040)
>2018-07-11 03:34:11.791333 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213441 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:11.966239 7f29149e4700  1 mon.magna102@0(leader).log v18041 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18041)
>2018-07-11 03:34:17.794007 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213444 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:17.966600 7f29149e4700  1 mon.magna102@0(leader).log v18042 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18042)
>2018-07-11 03:34:25.797091 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213447 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:25.967271 7f29149e4700  1 mon.magna102@0(leader).log v18043 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18043)
>2018-07-11 03:34:31.799950 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213450 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:31.976278 7f29149e4700  1 mon.magna102@0(leader).log v18044 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18044)
>2018-07-11 03:34:36.934548 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213453 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:37.157275 7f29149e4700  1 mon.magna102@0(leader).log v18045 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18045)
>2018-07-11 03:34:41.934781 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213456 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:42.935857 7f29149e4700  1 mon.magna102@0(leader).log v18046 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18046)
>2018-07-11 03:34:47.806597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213459 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:47.953226 7f29149e4700  1 mon.magna102@0(leader).log v18047 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18047)
>2018-07-11 03:34:53.809115 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213462 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:34:53.962153 7f29149e4700  1 mon.magna102@0(leader).log v18048 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18048)
>2018-07-11 03:34:59.847013 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8786 MB, avail 861 GB
>2018-07-11 03:35:03.813301 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213465 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:03.964272 7f29149e4700  1 mon.magna102@0(leader).log v18049 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18049)
>2018-07-11 03:35:11.935938 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213468 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:12.956593 7f29149e4700  1 mon.magna102@0(leader).log v18050 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18050)
>2018-07-11 03:35:16.936156 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213471 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:17.057157 7f29149e4700  1 mon.magna102@0(leader).log v18051 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18051)
>2018-07-11 03:35:21.936413 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213474 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:22.957775 7f29149e4700  1 mon.magna102@0(leader).log v18052 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18052)
>2018-07-11 03:35:26.936634 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213477 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:27.058283 7f29149e4700  1 mon.magna102@0(leader).log v18053 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18053)
>2018-07-11 03:35:33.826029 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213480 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:33.984026 7f29149e4700  1 mon.magna102@0(leader).log v18054 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18054)
>2018-07-11 03:35:43.830196 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213483 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:44.026621 7f29149e4700  1 mon.magna102@0(leader).log v18055 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18055)
>2018-07-11 03:35:49.832620 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213486 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:50.002220 7f29149e4700  1 mon.magna102@0(leader).log v18056 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18056)
>2018-07-11 03:35:55.835234 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213489 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:35:56.011137 7f29149e4700  1 mon.magna102@0(leader).log v18057 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18057)
>2018-07-11 03:35:59.879166 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8786 MB, avail 861 GB
>2018-07-11 03:36:01.937885 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213492 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:03.004663 7f29149e4700  1 mon.magna102@0(leader).log v18058 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18058)
>2018-07-11 03:36:07.840254 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213495 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:08.005319 7f29149e4700  1 mon.magna102@0(leader).log v18059 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18059)
>2018-07-11 03:36:16.938535 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213498 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:17.056013 7f29149e4700  1 mon.magna102@0(leader).log v18060 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18060)
>2018-07-11 03:36:21.948318 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213501 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:21.982239 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10097. Immutable memtables: 0.
>
>2018-07-11 03:36:21.982282 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 647] Syncing log #10094
>2018-07-11 03:36:21.998407 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:36:21.982272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:36:21.998419 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 647] Flushing memtable with next log file: 10097
>
>2018-07-11 03:36:21.998434 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280181998426, "job": 647, "event": "flush_started", "num_memtables": 1, "num_entries": 1631, "num_deletes": 251, "memory_usage": 3025792}
>2018-07-11 03:36:21.998437 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 647] Level-0 flush table #10098: started
>2018-07-11 03:36:22.065736 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280182065721, "cf_name": "default", "job": 647, "event": "table_file_creation", "file_number": 10098, "file_size": 2917379, "table_properties": {"data_size": 2893870, "index_size": 12750, "filter_size": 9835, "raw_key_size": 14082, "raw_average_key_size": 20, "raw_value_size": 2875207, "raw_average_value_size": 4197, "num_data_blocks": 384, "num_entries": 685, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 03:36:22.065747 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 647] Level-0 flush table #10098: 2917379 bytes OK
>2018-07-11 03:36:22.115737 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.082204) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10098 started
>2018-07-11 03:36:22.115745 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.115700) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10098: memtable #1 done
>2018-07-11 03:36:22.115748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.115714) EVENT_LOG_v1 {"time_micros": 1531280182115708, "job": 647, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:36:22.115763 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.115727) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:36:22.115770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 647] Try to delete WAL files size 2992900, prev total WAL file size 2992900, number of live WAL files 2.
>
>2018-07-11 03:36:22.116622 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:36:22.116815 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 03:36:22.116822 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 42209.0 total, 875.4 interval
>Cumulative writes: 59K writes, 286K keys, 59K commit groups, 1.0 writes per commit group, ingest: 0.48 GB, 0.01 MB/s
>Cumulative WAL: 59K writes, 59K syncs, 1.00 writes per sync, written: 0.48 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1158 writes, 5682 keys, 1158 commit groups, 1.0 writes per commit group, ingest: 8.65 MB, 0.01 MB/s
>Interval WAL: 1158 writes, 1158 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.78 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        27       324    0.083       0      0
>  L1      1/1   12.76 MB   0.0      5.1     0.4      4.7       4.7     -0.0       0.0  10.9     72.4     66.3        72       322    0.223   1137K   133K
> Sum      2/2   15.54 MB   0.0      5.1     0.4      4.7       5.1      0.4       0.0  11.9     52.7     52.7        99       646    0.153   1137K   133K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.5     55.6     55.6         1        10    0.144     19K   2917
>Uptime(secs): 42209.0 total, 42209.0 interval
>Flush(GB): cumulative 0.429, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.09 GB write, 0.12 MB/s write, 5.09 GB read, 0.12 MB/s read, 98.9 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.4 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.78 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        27       324    0.083       0      0
>  L1      1/1   12.76 MB   0.0      5.1     0.4      4.7       4.7     -0.0       0.0  10.9     72.4     66.3        72       322    0.223   1137K   133K
> Sum      2/2   15.54 MB   0.0      5.1     0.4      4.7       5.1      0.4       0.0  11.9     52.7     52.7        99       646    0.153   1137K   133K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 42209.0 total, 42209.0 interval
>Flush(GB): cumulative 0.429, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.09 GB write, 0.12 MB/s write, 5.09 GB read, 0.12 MB/s read, 98.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 03:36:22.116842 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.116836) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 03:36:22.116847 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 648] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:36:22.116853 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 647 Base level 0, inputs: [10098(2849KB)], [10096(12MB)]
>
>2018-07-11 03:36:22.116865 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280182116856, "job": 648, "event": "compaction_started", "files_L0": [10098], "files_L1": [10096], "score": -1, "input_data_size": 16294479}
>2018-07-11 03:36:22.337931 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 648] Generated table #10099: 3063 keys, 14299941 bytes
>2018-07-11 03:36:22.337978 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280182337951, "cf_name": "default", "job": 648, "event": "table_file_creation", "file_number": 10099, "file_size": 14299941, "table_properties": {"data_size": 14188508, "index_size": 63936, "filter_size": 46570, "raw_key_size": 61902, "raw_average_key_size": 20, "raw_value_size": 14099947, "raw_average_value_size": 4603, "num_data_blocks": 1867, "num_entries": 3063, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:36:22.374624 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 648] Compacted 1@0 + 1@1 files to L1 => 14299941 bytes
>2018-07-11 03:36:22.408165 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.408100) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 63.2 rd, 55.5 wr, level 1, files in(1, 1) out(1) MB in(2.8, 12.8) out(13.6), read-write-amplify(10.5) write-amplify(4.9) OK, records in: 3573, records dropped: 510
>
>2018-07-11 03:36:22.408174 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:36:22.408121) EVENT_LOG_v1 {"time_micros": 1531280182408112, "job": 648, "event": "compaction_finished", "compaction_time_micros": 257742, "output_level": 1, "num_output_files": 1, "total_output_size": 14299941, "num_input_records": 3573, "num_output_records": 3063, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:36:22.408831 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280182408827, "job": 648, "event": "table_file_deletion", "file_number": 10098}
>2018-07-11 03:36:22.411282 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280182411279, "job": 648, "event": "table_file_deletion", "file_number": 10096}
>2018-07-11 03:36:22.983802 7f29149e4700  1 mon.magna102@0(leader).log v18061 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18061)
>2018-07-11 03:36:26.948535 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213504 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:27.067492 7f29149e4700  1 mon.magna102@0(leader).log v18062 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18062)
>2018-07-11 03:36:31.948799 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213507 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:32.984815 7f29149e4700  1 mon.magna102@0(leader).log v18063 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18063)
>2018-07-11 03:36:39.853667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213510 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:40.003535 7f29149e4700  1 mon.magna102@0(leader).log v18064 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18064)
>2018-07-11 03:36:46.949398 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213513 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:47.070760 7f29149e4700  1 mon.magna102@0(leader).log v18065 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18065)
>2018-07-11 03:36:53.859606 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213516 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:54.029988 7f29149e4700  1 mon.magna102@0(leader).log v18066 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18066)
>2018-07-11 03:36:59.862111 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213519 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:36:59.896956 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:37:00.013996 7f29149e4700  1 mon.magna102@0(leader).log v18067 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18067)
>2018-07-11 03:37:07.865640 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213522 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:08.014737 7f29149e4700  1 mon.magna102@0(leader).log v18068 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18068)
>2018-07-11 03:37:16.950429 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213525 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:17.073805 7f29149e4700  1 mon.magna102@0(leader).log v18069 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18069)
>2018-07-11 03:37:21.950667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213528 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:23.007739 7f29149e4700  1 mon.magna102@0(leader).log v18070 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18070)
>2018-07-11 03:37:26.950895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213531 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:27.074916 7f29149e4700  1 mon.magna102@0(leader).log v18071 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18071)
>2018-07-11 03:37:33.876619 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213534 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:34.050741 7f29149e4700  1 mon.magna102@0(leader).log v18072 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18072)
>2018-07-11 03:37:39.879013 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213537 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:40.035876 7f29149e4700  1 mon.magna102@0(leader).log v18073 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18073)
>2018-07-11 03:37:46.951650 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213540 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:47.069828 7f29149e4700  1 mon.magna102@0(leader).log v18074 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18074)
>2018-07-11 03:37:51.951889 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213543 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:52.070483 7f29149e4700  1 mon.magna102@0(leader).log v18075 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18075)
>2018-07-11 03:37:57.886970 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213546 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:37:58.037734 7f29149e4700  1 mon.magna102@0(leader).log v18076 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18076)
>2018-07-11 03:37:59.921045 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:38:06.952479 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213549 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:07.021796 7f29149e4700  1 mon.magna102@0(leader).log v18077 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18077)
>2018-07-11 03:38:16.952873 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213552 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:17.031015 7f29149e4700  1 mon.magna102@0(leader).log v18078 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18078)
>2018-07-11 03:38:21.953118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213555 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:22.073462 7f29149e4700  1 mon.magna102@0(leader).log v18079 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18079)
>2018-07-11 03:38:26.953378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213558 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:27.024065 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10100. Immutable memtables: 0.
>
>2018-07-11 03:38:27.024109 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 649] Syncing log #10097
>2018-07-11 03:38:27.040651 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.024099) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:38:27.040662 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 649] Flushing memtable with next log file: 10100
>
>2018-07-11 03:38:27.040689 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307040681, "job": 649, "event": "flush_started", "num_memtables": 1, "num_entries": 841, "num_deletes": 250, "memory_usage": 1308784}
>2018-07-11 03:38:27.040692 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 649] Level-0 flush table #10101: started
>2018-07-11 03:38:27.116418 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307116403, "cf_name": "default", "job": 649, "event": "table_file_creation", "file_number": 10101, "file_size": 876832, "table_properties": {"data_size": 869039, "index_size": 3523, "filter_size": 3351, "raw_key_size": 8260, "raw_average_key_size": 21, "raw_value_size": 861228, "raw_average_value_size": 2242, "num_data_blocks": 109, "num_entries": 384, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 03:38:27.116428 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 649] Level-0 flush table #10101: 876832 bytes OK
>2018-07-11 03:38:27.149844 7f29149e4700  1 mon.magna102@0(leader).log v18080 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18080)
>2018-07-11 03:38:27.183059 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.149607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10101 started
>2018-07-11 03:38:27.183068 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.183021) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10101: memtable #1 done
>2018-07-11 03:38:27.183073 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.183037) EVENT_LOG_v1 {"time_micros": 1531280307183030, "job": 649, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:38:27.183075 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.183049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:38:27.183081 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 649] Try to delete WAL files size 1290925, prev total WAL file size 1309625, number of live WAL files 2.
>
>2018-07-11 03:38:27.183476 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:38:27.183557 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.183549) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 03:38:27.183569 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 650] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:38:27.183576 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 649 Base level 0, inputs: [10101(856KB)], [10099(13MB)]
>
>2018-07-11 03:38:27.183590 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307183580, "job": 650, "event": "compaction_started", "files_L0": [10101], "files_L1": [10099], "score": -1, "input_data_size": 15176773}
>2018-07-11 03:38:27.351699 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 650] Generated table #10102: 2995 keys, 13846147 bytes
>2018-07-11 03:38:27.351748 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307351721, "cf_name": "default", "job": 650, "event": "table_file_creation", "file_number": 10102, "file_size": 13846147, "table_properties": {"data_size": 13739475, "index_size": 60786, "filter_size": 44959, "raw_key_size": 60185, "raw_average_key_size": 20, "raw_value_size": 13654168, "raw_average_value_size": 4558, "num_data_blocks": 1781, "num_entries": 2995, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:38:27.384899 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 650] Compacted 1@0 + 1@1 files to L1 => 13846147 bytes
>2018-07-11 03:38:27.418385 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.418317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.4 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(0.8, 13.6) out(13.2), read-write-amplify(33.1) write-amplify(15.8) OK, records in: 3447, records dropped: 452
>
>2018-07-11 03:38:27.418393 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.418338) EVENT_LOG_v1 {"time_micros": 1531280307418328, "job": 650, "event": "compaction_finished", "compaction_time_micros": 201295, "output_level": 1, "num_output_files": 1, "total_output_size": 13846147, "num_input_records": 3447, "num_output_records": 2995, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:38:27.418649 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307418645, "job": 650, "event": "table_file_deletion", "file_number": 10101}
>2018-07-11 03:38:27.421472 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307421469, "job": 650, "event": "table_file_deletion", "file_number": 10099}
>2018-07-11 03:38:27.421791 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10103. Immutable memtables: 0.
>
>2018-07-11 03:38:27.421907 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 651] Syncing log #10100
>2018-07-11 03:38:27.435445 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.421897) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:38:27.435458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 651] Flushing memtable with next log file: 10103
>
>2018-07-11 03:38:27.435472 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307435465, "job": 651, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21064}
>2018-07-11 03:38:27.435475 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 651] Level-0 flush table #10104: started
>2018-07-11 03:38:27.477531 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307477519, "cf_name": "default", "job": 651, "event": "table_file_creation", "file_number": 10104, "file_size": 19819, "table_properties": {"data_size": 18749, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18526, "raw_average_value_size": 2315, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:38:27.477541 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 651] Level-0 flush table #10104: 19819 bytes OK
>2018-07-11 03:38:27.527595 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.494085) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10104 started
>2018-07-11 03:38:27.527604 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.527561) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10104: memtable #1 done
>2018-07-11 03:38:27.527607 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.527576) EVENT_LOG_v1 {"time_micros": 1531280307527569, "job": 651, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:38:27.527621 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.527587) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:38:27.527626 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 651] Try to delete WAL files size 18700, prev total WAL file size 18700, number of live WAL files 2.
>
>2018-07-11 03:38:27.527724 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:38:27.527771 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.527765) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 03:38:27.527780 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 652] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:38:27.527786 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 651 Base level 0, inputs: [10104(19KB)], [10102(13MB)]
>
>2018-07-11 03:38:27.527799 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307527789, "job": 652, "event": "compaction_started", "files_L0": [10104], "files_L1": [10102], "score": -1, "input_data_size": 13865966}
>2018-07-11 03:38:27.694916 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 652] Generated table #10105: 2998 keys, 13864849 bytes
>2018-07-11 03:38:27.694962 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307694947, "cf_name": "default", "job": 652, "event": "table_file_creation", "file_number": 10105, "file_size": 13864849, "table_properties": {"data_size": 13758051, "index_size": 60855, "filter_size": 45016, "raw_key_size": 60246, "raw_average_key_size": 20, "raw_value_size": 13672654, "raw_average_value_size": 4560, "num_data_blocks": 1783, "num_entries": 2998, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:38:27.728213 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 652] Compacted 1@0 + 1@1 files to L1 => 13864849 bytes
>2018-07-11 03:38:27.761717 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.761645) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.2 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.2) out(13.2), read-write-amplify(1399.2) write-amplify(699.6) OK, records in: 3003, records dropped: 5
>
>2018-07-11 03:38:27.761726 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:38:27.761667) EVENT_LOG_v1 {"time_micros": 1531280307761657, "job": 652, "event": "compaction_finished", "compaction_time_micros": 200398, "output_level": 1, "num_output_files": 1, "total_output_size": 13864849, "num_input_records": 3003, "num_output_records": 2998, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:38:27.761822 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307761817, "job": 652, "event": "table_file_deletion", "file_number": 10104}
>2018-07-11 03:38:27.764441 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280307764439, "job": 652, "event": "table_file_deletion", "file_number": 10102}
>2018-07-11 03:38:31.990693 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213561 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:33.037796 7f29149e4700  1 mon.magna102@0(leader).log v18081 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18081)
>2018-07-11 03:38:37.904539 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213564 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:38.088515 7f29149e4700  1 mon.magna102@0(leader).log v18082 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18082)
>2018-07-11 03:38:46.991326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213567 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:47.114211 7f29149e4700  1 mon.magna102@0(leader).log v18083 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18083)
>2018-07-11 03:38:51.991562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213570 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:53.048370 7f29149e4700  1 mon.magna102@0(leader).log v18084 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18084)
>2018-07-11 03:38:56.991790 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213573 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:38:57.115531 7f29149e4700  1 mon.magna102@0(leader).log v18085 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18085)
>2018-07-11 03:38:59.949012 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:39:01.992019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213576 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:03.049503 7f29149e4700  1 mon.magna102@0(leader).log v18086 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18086)
>2018-07-11 03:39:07.917281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213579 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:08.093047 7f29149e4700  1 mon.magna102@0(leader).log v18087 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18087)
>2018-07-11 03:39:13.919719 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213582 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:14.093662 7f29149e4700  1 mon.magna102@0(leader).log v18088 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18088)
>2018-07-11 03:39:19.922112 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213585 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:20.094254 7f29149e4700  1 mon.magna102@0(leader).log v18089 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18089)
>2018-07-11 03:39:27.925597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213588 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:28.103404 7f29149e4700  1 mon.magna102@0(leader).log v18090 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18090)
>2018-07-11 03:39:33.928214 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213591 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:34.104014 7f29149e4700  1 mon.magna102@0(leader).log v18091 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18091)
>2018-07-11 03:39:41.993475 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213594 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:42.113115 7f29149e4700  1 mon.magna102@0(leader).log v18092 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18092)
>2018-07-11 03:39:46.993710 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213597 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:47.063556 7f29149e4700  1 mon.magna102@0(leader).log v18093 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18093)
>2018-07-11 03:39:53.936587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213600 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:39:54.089339 7f29149e4700  1 mon.magna102@0(leader).log v18094 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18094)
>2018-07-11 03:39:59.972915 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:40:01.994298 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213603 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:02.116683 7f29149e4700  1 mon.magna102@0(leader).log v18095 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18095)
>2018-07-11 03:40:06.994627 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213606 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:07.100384 7f29149e4700  1 mon.magna102@0(leader).log v18096 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18096)
>2018-07-11 03:40:16.995019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213609 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:17.076355 7f29149e4700  1 mon.magna102@0(leader).log v18097 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18097)
>2018-07-11 03:40:21.995247 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213612 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:22.118673 7f29149e4700  1 mon.magna102@0(leader).log v18098 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18098)
>2018-07-11 03:40:26.995486 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213615 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:27.085867 7f29149e4700  1 mon.magna102@0(leader).log v18099 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18099)
>2018-07-11 03:40:36.995891 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213618 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:37.095134 7f29149e4700  1 mon.magna102@0(leader).log v18100 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18100)
>2018-07-11 03:40:41.996081 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213621 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:42.120866 7f29149e4700  1 mon.magna102@0(leader).log v18101 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18101)
>2018-07-11 03:40:46.996278 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213624 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:47.121202 7f29149e4700  1 mon.magna102@0(leader).log v18102 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18102)
>2018-07-11 03:40:51.996630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213627 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:52.130326 7f29149e4700  1 mon.magna102@0(leader).log v18103 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18103)
>2018-07-11 03:40:56.996845 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213630 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:40:57.107004 7f29149e4700  1 mon.magna102@0(leader).log v18104 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18104)
>2018-07-11 03:40:59.998842 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:41:03.966515 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213633 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:04.124473 7f29149e4700  1 mon.magna102@0(leader).log v18105 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18105)
>2018-07-11 03:41:11.999855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213636 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:12.116904 7f29149e4700  1 mon.magna102@0(leader).log v18106 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18106)
>2018-07-11 03:41:17.000074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213639 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:17.109161 7f29149e4700  1 mon.magna102@0(leader).log v18107 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18107)
>2018-07-11 03:41:27.009517 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213642 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:27.110075 7f29149e4700  1 mon.magna102@0(leader).log v18108 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18108)
>2018-07-11 03:41:32.010353 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213645 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:32.127383 7f29149e4700  1 mon.magna102@0(leader).log v18109 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18109)
>2018-07-11 03:41:37.010581 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213648 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:37.111192 7f29149e4700  1 mon.magna102@0(leader).log v18110 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18110)
>2018-07-11 03:41:42.019812 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213651 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:42.136727 7f29149e4700  1 mon.magna102@0(leader).log v18111 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18111)
>2018-07-11 03:41:47.985282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213654 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:48.145793 7f29149e4700  1 mon.magna102@0(leader).log v18112 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18112)
>2018-07-11 03:41:57.022321 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213657 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:41:57.122733 7f29149e4700  1 mon.magna102@0(leader).log v18113 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18113)
>2018-07-11 03:42:00.022877 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:42:02.031370 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213660 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:02.148437 7f29149e4700  1 mon.magna102@0(leader).log v18114 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18114)
>2018-07-11 03:42:07.031596 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213663 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:07.140550 7f29149e4700  1 mon.magna102@0(leader).log v18115 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18115)
>2018-07-11 03:42:17.032679 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213666 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:17.158080 7f29149e4700  1 mon.magna102@0(leader).log v18116 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18116)
>2018-07-11 03:42:22.041628 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213669 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:22.175441 7f29149e4700  1 mon.magna102@0(leader).log v18117 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18117)
>2018-07-11 03:42:27.041843 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213672 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:27.167709 7f29149e4700  1 mon.magna102@0(leader).log v18118 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18118)
>2018-07-11 03:42:32.042923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213675 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:32.176696 7f29149e4700  1 mon.magna102@0(leader).log v18119 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18119)
>2018-07-11 03:42:37.043150 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213678 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:37.143819 7f29149e4700  1 mon.magna102@0(leader).log v18120 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18120)
>2018-07-11 03:42:40.136229 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10106. Immutable memtables: 0.
>
>2018-07-11 03:42:40.136274 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 653] Syncing log #10103
>2018-07-11 03:42:40.152755 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.136264) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:42:40.152767 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 653] Flushing memtable with next log file: 10106
>
>2018-07-11 03:42:40.152794 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560152785, "job": 653, "event": "flush_started", "num_memtables": 1, "num_entries": 1465, "num_deletes": 251, "memory_usage": 2678984}
>2018-07-11 03:42:40.152797 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 653] Level-0 flush table #10107: started
>2018-07-11 03:42:40.211583 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560211556, "cf_name": "default", "job": 653, "event": "table_file_creation", "file_number": 10107, "file_size": 2583150, "table_properties": {"data_size": 2562274, "index_size": 11188, "filter_size": 8764, "raw_key_size": 13021, "raw_average_key_size": 20, "raw_value_size": 2545425, "raw_average_value_size": 4014, "num_data_blocks": 338, "num_entries": 634, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 03:42:40.211595 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 653] Level-0 flush table #10107: 2583150 bytes OK
>2018-07-11 03:42:40.271135 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.228071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10107 started
>2018-07-11 03:42:40.271144 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.271096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10107: memtable #1 done
>2018-07-11 03:42:40.271146 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.271111) EVENT_LOG_v1 {"time_micros": 1531280560271104, "job": 653, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:42:40.271148 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.271124) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:42:40.271155 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 653] Try to delete WAL files size 2649016, prev total WAL file size 2649016, number of live WAL files 2.
>
>2018-07-11 03:42:40.271836 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:42:40.271888 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.271880) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 03:42:40.271897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 654] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:42:40.271904 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 653 Base level 0, inputs: [10107(2522KB)], [10105(13MB)]
>
>2018-07-11 03:42:40.271917 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560271907, "job": 654, "event": "compaction_started", "files_L0": [10107], "files_L1": [10105], "score": -1, "input_data_size": 16447999}
>2018-07-11 03:42:40.446846 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 654] Generated table #10108: 3122 keys, 14453742 bytes
>2018-07-11 03:42:40.446896 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560446879, "cf_name": "default", "job": 654, "event": "table_file_creation", "file_number": 10108, "file_size": 14453742, "table_properties": {"data_size": 14341749, "index_size": 64052, "filter_size": 47014, "raw_key_size": 63020, "raw_average_key_size": 20, "raw_value_size": 14252275, "raw_average_value_size": 4565, "num_data_blocks": 1869, "num_entries": 3122, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:42:40.480030 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 654] Compacted 1@0 + 1@1 files to L1 => 14453742 bytes
>2018-07-11 03:42:40.513561 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.513492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.0 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.8), read-write-amplify(12.0) write-amplify(5.6) OK, records in: 3632, records dropped: 510
>
>2018-07-11 03:42:40.513570 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:42:40.513514) EVENT_LOG_v1 {"time_micros": 1531280560513504, "job": 654, "event": "compaction_finished", "compaction_time_micros": 208096, "output_level": 1, "num_output_files": 1, "total_output_size": 14453742, "num_input_records": 3632, "num_output_records": 3122, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:42:40.514162 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560514141, "job": 654, "event": "table_file_deletion", "file_number": 10107}
>2018-07-11 03:42:40.516923 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280560516908, "job": 654, "event": "table_file_deletion", "file_number": 10105}
>2018-07-11 03:42:47.047528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213681 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:47.148165 7f29149e4700  1 mon.magna102@0(leader).log v18121 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18121)
>2018-07-11 03:42:52.048467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213684 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:52.173803 7f29149e4700  1 mon.magna102@0(leader).log v18122 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18122)
>2018-07-11 03:42:57.048696 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213687 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:42:57.174282 7f29149e4700  1 mon.magna102@0(leader).log v18123 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18123)
>2018-07-11 03:43:00.057885 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:43:06.018460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213693 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:06.191977 7f29149e4700  1 mon.magna102@0(leader).log v18124 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18124)
>2018-07-11 03:43:14.022037 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213696 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:14.192687 7f29149e4700  1 mon.magna102@0(leader).log v18125 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18125)
>2018-07-11 03:43:22.025402 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213699 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:22.176749 7f29149e4700  1 mon.magna102@0(leader).log v18126 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18126)
>2018-07-11 03:43:27.059922 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213702 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:27.160529 7f29149e4700  1 mon.magna102@0(leader).log v18127 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18127)
>2018-07-11 03:43:34.030403 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213705 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:34.187539 7f29149e4700  1 mon.magna102@0(leader).log v18128 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18128)
>2018-07-11 03:43:35.929326 7f29149e4700  1 mon.magna102@0(leader).log v18129 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18129)
>2018-07-11 03:43:42.063164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213708 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:42.180169 7f29149e4700  1 mon.magna102@0(leader).log v18130 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18130)
>2018-07-11 03:43:48.036416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213711 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:48.189106 7f29149e4700  1 mon.magna102@0(leader).log v18131 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18131)
>2018-07-11 03:43:55.931263 7f29149e4700  1 mon.magna102@0(leader).log v18132 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18132)
>2018-07-11 03:43:57.072761 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213714 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:43:58.140172 7f29149e4700  1 mon.magna102@0(leader).log v18133 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18133)
>2018-07-11 03:44:00.073482 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:44:02.073635 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213717 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:02.190702 7f29149e4700  1 mon.magna102@0(leader).log v18134 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18134)
>2018-07-11 03:44:07.073855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213720 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:07.182809 7f29149e4700  1 mon.magna102@0(leader).log v18135 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18135)
>2018-07-11 03:44:15.933561 7f29149e4700  1 mon.magna102@0(leader).log v18136 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18136)
>2018-07-11 03:44:17.083385 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213723 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:18.134142 7f29149e4700  1 mon.magna102@0(leader).log v18137 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18137)
>2018-07-11 03:44:22.084266 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213726 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:22.201290 7f29149e4700  1 mon.magna102@0(leader).log v18138 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18138)
>2018-07-11 03:44:25.944403 7f29149e4700  1 mon.magna102@0(leader).log v18139 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18139)
>2018-07-11 03:44:28.027919 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213729 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:28.220071 7f29149e4700  1 mon.magna102@0(leader).log v18140 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18140)
>2018-07-11 03:44:35.945528 7f29149e4700  1 mon.magna102@0(leader).log v18141 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18141)
>2018-07-11 03:44:37.095386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213732 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:38.196118 7f29149e4700  1 mon.magna102@0(leader).log v18142 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18142)
>2018-07-11 03:44:42.096242 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213735 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:42.229909 7f29149e4700  1 mon.magna102@0(leader).log v18143 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18143)
>2018-07-11 03:44:47.096455 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213738 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:47.188733 7f29149e4700  1 mon.magna102@0(leader).log v18144 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18144)
>2018-07-11 03:44:57.097615 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213741 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:44:57.198139 7f29149e4700  1 mon.magna102@0(leader).log v18145 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18145)
>2018-07-11 03:45:00.098290 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:45:02.098475 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213744 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:02.215517 7f29149e4700  1 mon.magna102@0(leader).log v18146 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18146)
>2018-07-11 03:45:07.098642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213747 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:07.199220 7f29149e4700  1 mon.magna102@0(leader).log v18147 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18147)
>2018-07-11 03:45:12.107914 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213750 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:12.224946 7f29149e4700  1 mon.magna102@0(leader).log v18148 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18148)
>2018-07-11 03:45:17.108139 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213753 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:17.209972 7f29149e4700  1 mon.magna102@0(leader).log v18149 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18149)
>2018-07-11 03:45:24.077005 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213756 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:24.227397 7f29149e4700  1 mon.magna102@0(leader).log v18150 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18150)
>2018-07-11 03:45:32.119460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213759 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:32.236501 7f29149e4700  1 mon.magna102@0(leader).log v18151 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18151)
>2018-07-11 03:45:37.119690 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213762 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:37.220279 7f29149e4700  1 mon.magna102@0(leader).log v18152 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18152)
>2018-07-11 03:45:44.085533 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213765 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:44.246056 7f29149e4700  1 mon.magna102@0(leader).log v18153 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18153)
>2018-07-11 03:45:52.129772 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213768 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:52.246802 7f29149e4700  1 mon.magna102@0(leader).log v18154 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18154)
>2018-07-11 03:45:57.130002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213771 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:45:57.230584 7f29149e4700  1 mon.magna102@0(leader).log v18155 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18155)
>2018-07-11 03:46:00.130750 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:46:02.130912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213774 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:02.256295 7f29149e4700  1 mon.magna102@0(leader).log v18156 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18156)
>2018-07-11 03:46:07.131123 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213777 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:07.256830 7f29149e4700  1 mon.magna102@0(leader).log v18157 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18157)
>2018-07-11 03:46:14.098334 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213780 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:14.258817 7f29149e4700  1 mon.magna102@0(leader).log v18158 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18158)
>2018-07-11 03:46:22.117666 7f29149e4700  1 mon.magna102@0(leader).log v18159 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18159)
>2018-07-11 03:46:22.151074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213783 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:23.184784 7f29149e4700  1 mon.magna102@0(leader).log v18160 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18160)
>2018-07-11 03:46:28.104436 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213786 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:28.252081 7f29149e4700  1 mon.magna102@0(leader).log v18161 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18161)
>2018-07-11 03:46:32.118972 7f29149e4700  1 mon.magna102@0(leader).log v18162 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18162)
>2018-07-11 03:46:36.107595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213792 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:36.261352 7f29149e4700  1 mon.magna102@0(leader).log v18163 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18163)
>2018-07-11 03:46:44.111164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213795 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:44.295501 7f29149e4700  1 mon.magna102@0(leader).log v18164 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18164)
>2018-07-11 03:46:47.237710 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10109. Immutable memtables: 0.
>
>2018-07-11 03:46:47.237754 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 655] Syncing log #10106
>2018-07-11 03:46:47.254218 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.237744) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:46:47.254230 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 655] Flushing memtable with next log file: 10109
>
>2018-07-11 03:46:47.254256 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807254248, "job": 655, "event": "flush_started", "num_memtables": 1, "num_entries": 1468, "num_deletes": 250, "memory_usage": 2672072}
>2018-07-11 03:46:47.254259 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 655] Level-0 flush table #10110: started
>2018-07-11 03:46:47.313101 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807313087, "cf_name": "default", "job": 655, "event": "table_file_creation", "file_number": 10110, "file_size": 1779641, "table_properties": {"data_size": 1765395, "index_size": 7167, "filter_size": 6159, "raw_key_size": 11005, "raw_average_key_size": 21, "raw_value_size": 1753051, "raw_average_value_size": 3371, "num_data_blocks": 221, "num_entries": 520, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 03:46:47.313125 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 655] Level-0 flush table #10110: 1779641 bytes OK
>2018-07-11 03:46:47.363123 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.329616) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10110 started
>2018-07-11 03:46:47.363131 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.363087) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10110: memtable #1 done
>2018-07-11 03:46:47.363134 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.363102) EVENT_LOG_v1 {"time_micros": 1531280807363095, "job": 655, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:46:47.363148 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.363113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:46:47.363155 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 655] Try to delete WAL files size 2642745, prev total WAL file size 2642745, number of live WAL files 2.
>
>2018-07-11 03:46:47.363839 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:46:47.364021 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 03:46:47.364028 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 42834.2 total, 625.2 interval
>Cumulative writes: 60K writes, 290K keys, 60K commit groups, 1.0 writes per commit group, ingest: 0.49 GB, 0.01 MB/s
>Cumulative WAL: 60K writes, 60K syncs, 1.00 writes per sync, written: 0.49 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 838 writes, 3782 keys, 838 commit groups, 1.0 writes per commit group, ingest: 6.30 MB, 0.01 MB/s
>Interval WAL: 838 writes, 838 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.70 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        27       328    0.083       0      0
>  L1      1/1   13.78 MB   0.0      5.1     0.4      4.7       4.7     -0.0       0.0  10.9     72.4     66.3        73       326    0.223   1151K   135K
> Sum      2/2   15.48 MB   0.0      5.1     0.4      4.7       5.1      0.4       0.0  11.9     52.6     52.6       100       654    0.153   1151K   135K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.7     49.7     49.6         1         8    0.148     13K   1477
>Uptime(secs): 42834.2 total, 42834.2 interval
>Flush(GB): cumulative 0.434, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.14 GB write, 0.12 MB/s write, 5.14 GB read, 0.12 MB/s read, 100.0 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.70 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        27       328    0.083       0      0
>  L1      1/1   13.78 MB   0.0      5.1     0.4      4.7       4.7     -0.0       0.0  10.9     72.4     66.3        73       326    0.223   1151K   135K
> Sum      2/2   15.48 MB   0.0      5.1     0.4      4.7       5.1      0.4       0.0  11.9     52.6     52.6       100       654    0.153   1151K   135K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 42834.2 total, 42834.2 interval
>Flush(GB): cumulative 0.434, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.14 GB write, 0.12 MB/s write, 5.14 GB read, 0.12 MB/s read, 100.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 03:46:47.364052 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.364045) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 03:46:47.364057 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 656] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:46:47.364064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 655 Base level 0, inputs: [10110(1737KB)], [10108(13MB)]
>
>2018-07-11 03:46:47.364077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807364067, "job": 656, "event": "compaction_started", "files_L0": [10110], "files_L1": [10108], "score": -1, "input_data_size": 16233383}
>2018-07-11 03:46:47.572173 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 656] Generated table #10111: 3251 keys, 15319612 bytes
>2018-07-11 03:46:47.572219 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807572203, "cf_name": "default", "job": 656, "event": "table_file_creation", "file_number": 10111, "file_size": 15319612, "table_properties": {"data_size": 15202329, "index_size": 66785, "filter_size": 49571, "raw_key_size": 65390, "raw_average_key_size": 20, "raw_value_size": 15109189, "raw_average_value_size": 4647, "num_data_blocks": 1956, "num_entries": 3251, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:46:47.613638 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 656] Compacted 1@0 + 1@1 files to L1 => 15319612 bytes
>2018-07-11 03:46:47.647176 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.647110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 65.1 rd, 61.4 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.8) out(14.6), read-write-amplify(17.7) write-amplify(8.6) OK, records in: 3642, records dropped: 391
>
>2018-07-11 03:46:47.647196 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:46:47.647131) EVENT_LOG_v1 {"time_micros": 1531280807647122, "job": 656, "event": "compaction_finished", "compaction_time_micros": 249545, "output_level": 1, "num_output_files": 1, "total_output_size": 15319612, "num_input_records": 3642, "num_output_records": 3251, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:46:47.647633 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807647628, "job": 656, "event": "table_file_deletion", "file_number": 10110}
>2018-07-11 03:46:47.650313 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280807650310, "job": 656, "event": "table_file_deletion", "file_number": 10108}
>2018-07-11 03:46:47.650359 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:46:52.204268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213798 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:53.328451 7f29149e4700  1 mon.magna102@0(leader).log v18165 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18165)
>2018-07-11 03:46:57.204528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213801 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:46:57.324989 7f29149e4700  1 mon.magna102@0(leader).log v18166 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18166)
>2018-07-11 03:47:00.150159 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 03:47:04.119846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213804 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:04.267403 7f29149e4700  1 mon.magna102@0(leader).log v18167 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18167)
>2018-07-11 03:47:12.123268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213810 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:12.276554 7f29149e4700  1 mon.magna102@0(leader).log v18168 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18168)
>2018-07-11 03:47:24.128249 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213813 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:24.277710 7f29149e4700  1 mon.magna102@0(leader).log v18169 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18169)
>2018-07-11 03:47:32.131591 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213819 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:32.286670 7f29149e4700  1 mon.magna102@0(leader).log v18170 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18170)
>2018-07-11 03:47:42.135949 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213822 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:42.287656 7f29149e4700  1 mon.magna102@0(leader).log v18171 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18171)
>2018-07-11 03:47:47.206312 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213825 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:47.363083 7f29149e4700  1 mon.magna102@0(leader).log v18172 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18172)
>2018-07-11 03:47:54.141132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213828 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:47:54.306752 7f29149e4700  1 mon.magna102@0(leader).log v18173 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18173)
>2018-07-11 03:48:00.190408 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 03:48:02.206874 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213831 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:02.340991 7f29149e4700  1 mon.magna102@0(leader).log v18174 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18174)
>2018-07-11 03:48:07.207100 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213834 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:07.308098 7f29149e4700  1 mon.magna102@0(leader).log v18175 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18175)
>2018-07-11 03:48:17.207529 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213837 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:17.283983 7f29149e4700  1 mon.magna102@0(leader).log v18176 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18176)
>2018-07-11 03:48:22.207765 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213840 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:22.326392 7f29149e4700  1 mon.magna102@0(leader).log v18177 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18177)
>2018-07-11 03:48:27.208026 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213843 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:27.285175 7f29149e4700  1 mon.magna102@0(leader).log v18178 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18178)
>2018-07-11 03:48:32.208239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213846 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:32.327600 7f29149e4700  1 mon.magna102@0(leader).log v18179 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18179)
>2018-07-11 03:48:37.208458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213849 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:37.294698 7f29149e4700  1 mon.magna102@0(leader).log v18180 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18180)
>2018-07-11 03:48:47.208856 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213852 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:47.295601 7f29149e4700  1 mon.magna102@0(leader).log v18181 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18181)
>2018-07-11 03:48:52.209081 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213855 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:52.330834 7f29149e4700  1 mon.magna102@0(leader).log v18182 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18182)
>2018-07-11 03:48:54.298137 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10112. Immutable memtables: 0.
>
>2018-07-11 03:48:54.298188 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 657] Syncing log #10109
>2018-07-11 03:48:54.314706 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.298171) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:48:54.314718 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 657] Flushing memtable with next log file: 10112
>
>2018-07-11 03:48:54.314744 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934314736, "job": 657, "event": "flush_started", "num_memtables": 1, "num_entries": 848, "num_deletes": 251, "memory_usage": 1301144}
>2018-07-11 03:48:54.314747 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 657] Level-0 flush table #10113: started
>2018-07-11 03:48:54.365274 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934365260, "cf_name": "default", "job": 657, "event": "table_file_creation", "file_number": 10113, "file_size": 1250568, "table_properties": {"data_size": 1239567, "index_size": 5482, "filter_size": 4599, "raw_key_size": 9033, "raw_average_key_size": 20, "raw_value_size": 1229532, "raw_average_value_size": 2781, "num_data_blocks": 167, "num_entries": 442, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 03:48:54.365283 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 657] Level-0 flush table #10113: 1250568 bytes OK
>2018-07-11 03:48:54.415282 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.381785) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10113 started
>2018-07-11 03:48:54.415290 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.415246) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10113: memtable #1 done
>2018-07-11 03:48:54.415293 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.415261) EVENT_LOG_v1 {"time_micros": 1531280934415254, "job": 657, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:48:54.415307 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.415272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:48:54.415314 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 657] Try to delete WAL files size 1282803, prev total WAL file size 1282803, number of live WAL files 2.
>
>2018-07-11 03:48:54.415734 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:48:54.415797 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.415789) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 03:48:54.415803 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 658] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:48:54.415810 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 657 Base level 0, inputs: [10113(1221KB)], [10111(14MB)]
>
>2018-07-11 03:48:54.415823 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934415813, "job": 658, "event": "compaction_started", "files_L0": [10113], "files_L1": [10111], "score": -1, "input_data_size": 16570180}
>2018-07-11 03:48:54.590990 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 658] Generated table #10114: 3183 keys, 14573032 bytes
>2018-07-11 03:48:54.591039 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934591012, "cf_name": "default", "job": 658, "event": "table_file_creation", "file_number": 10114, "file_size": 14573032, "table_properties": {"data_size": 14460542, "index_size": 64168, "filter_size": 47395, "raw_key_size": 64176, "raw_average_key_size": 20, "raw_value_size": 14370135, "raw_average_value_size": 4514, "num_data_blocks": 1871, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:48:54.624188 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 658] Compacted 1@0 + 1@1 files to L1 => 14573032 bytes
>2018-07-11 03:48:54.657678 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.657611) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.5 rd, 69.9 wr, level 1, files in(1, 1) out(1) MB in(1.2, 14.6) out(13.9), read-write-amplify(24.9) write-amplify(11.7) OK, records in: 3693, records dropped: 510
>
>2018-07-11 03:48:54.657687 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:48:54.657633) EVENT_LOG_v1 {"time_micros": 1531280934657623, "job": 658, "event": "compaction_finished", "compaction_time_micros": 208342, "output_level": 1, "num_output_files": 1, "total_output_size": 14573032, "num_input_records": 3693, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:48:54.658009 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934658005, "job": 658, "event": "table_file_deletion", "file_number": 10113}
>2018-07-11 03:48:54.660992 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934660989, "job": 658, "event": "table_file_deletion", "file_number": 10111}
>2018-07-11 03:48:57.209315 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213858 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:48:57.300150 7f29149e4700  1 mon.magna102@0(leader).log v18183 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18183)
>2018-07-11 03:49:00.200298 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:49:04.171006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213861 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:04.325937 7f29149e4700  1 mon.magna102@0(leader).log v18184 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18184)
>2018-07-11 03:49:10.173406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213864 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:10.334974 7f29149e4700  1 mon.magna102@0(leader).log v18185 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18185)
>2018-07-11 03:49:16.175986 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213867 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:16.327218 7f29149e4700  1 mon.magna102@0(leader).log v18186 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18186)
>2018-07-11 03:49:22.178462 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213870 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:22.327815 7f29149e4700  1 mon.magna102@0(leader).log v18187 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18187)
>2018-07-11 03:49:28.181130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213873 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:28.328335 7f29149e4700  1 mon.magna102@0(leader).log v18188 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18188)
>2018-07-11 03:49:34.183651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213876 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:34.338566 7f29149e4700  1 mon.magna102@0(leader).log v18189 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18189)
>2018-07-11 03:49:42.187038 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213879 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:42.339299 7f29149e4700  1 mon.magna102@0(leader).log v18190 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18190)
>2018-07-11 03:49:48.189557 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213882 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:48.339953 7f29149e4700  1 mon.magna102@0(leader).log v18191 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18191)
>2018-07-11 03:49:57.231919 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213885 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:49:57.340709 7f29149e4700  1 mon.magna102@0(leader).log v18192 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18192)
>2018-07-11 03:50:00.232522 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:50:07.241864 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213888 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:07.359016 7f29149e4700  1 mon.magna102@0(leader).log v18193 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18193)
>2018-07-11 03:50:12.242685 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213891 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:12.376354 7f29149e4700  1 mon.magna102@0(leader).log v18194 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18194)
>2018-07-11 03:50:17.242895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213894 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:17.343447 7f29149e4700  1 mon.magna102@0(leader).log v18195 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18195)
>2018-07-11 03:50:22.243130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213897 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:22.360883 7f29149e4700  1 mon.magna102@0(leader).log v18196 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18196)
>2018-07-11 03:50:28.206604 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213900 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:28.354383 7f29149e4700  1 mon.magna102@0(leader).log v18197 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18197)
>2018-07-11 03:50:36.210019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213906 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:36.363578 7f29149e4700  1 mon.magna102@0(leader).log v18198 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18198)
>2018-07-11 03:50:44.213469 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213909 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:44.364306 7f29149e4700  1 mon.magna102@0(leader).log v18199 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18199)
>2018-07-11 03:50:52.217008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213912 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:50:52.373461 7f29149e4700  1 mon.magna102@0(leader).log v18200 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18200)
>2018-07-11 03:50:57.357282 7f29149e4700  1 mon.magna102@0(leader).log v18201 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18201)
>2018-07-11 03:50:59.524259 7f29149e4700  1 mon.magna102@0(leader).log v18202 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18202)
>2018-07-11 03:51:00.232689 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:51:00.524286 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213915 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:00.682980 7f29149e4700  1 mon.magna102@0(leader).log v18203 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18203)
>2018-07-11 03:51:07.258135 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213918 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:07.358424 7f29149e4700  1 mon.magna102@0(leader).log v18204 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18204)
>2018-07-11 03:51:12.258872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213921 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:12.384260 7f29149e4700  1 mon.magna102@0(leader).log v18205 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18205)
>2018-07-11 03:51:17.259078 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213924 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:17.376370 7f29149e4700  1 mon.magna102@0(leader).log v18206 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18206)
>2018-07-11 03:51:27.269740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213927 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:27.395146 7f29149e4700  1 mon.magna102@0(leader).log v18207 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18207)
>2018-07-11 03:51:32.270449 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213930 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:32.404165 7f29149e4700  1 mon.magna102@0(leader).log v18208 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18208)
>2018-07-11 03:51:37.270659 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213933 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:37.396275 7f29149e4700  1 mon.magna102@0(leader).log v18209 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18209)
>2018-07-11 03:51:39.563216 7f29149e4700  1 mon.magna102@0(leader).log v18210 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18210)
>2018-07-11 03:51:47.271932 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213936 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:47.372336 7f29149e4700  1 mon.magna102@0(leader).log v18211 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18211)
>2018-07-11 03:51:52.272654 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213939 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:52.389688 7f29149e4700  1 mon.magna102@0(leader).log v18212 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18212)
>2018-07-11 03:51:57.272864 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213942 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:51:57.373462 7f29149e4700  1 mon.magna102@0(leader).log v18213 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18213)
>2018-07-11 03:51:59.515413 7f29149e4700  1 mon.magna102@0(leader).log v18214 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18214)
>2018-07-11 03:52:00.232873 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:52:07.282446 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213945 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:07.382973 7f29149e4700  1 mon.magna102@0(leader).log v18215 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18215)
>2018-07-11 03:52:12.283250 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213948 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:12.400342 7f29149e4700  1 mon.magna102@0(leader).log v18216 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18216)
>2018-07-11 03:52:17.283460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213954 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:17.385328 7f29149e4700  1 mon.magna102@0(leader).log v18217 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18217)
>2018-07-11 03:52:23.852522 7f29149e4700  1 mon.magna102@0(leader).log v18218 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18218)
>2018-07-11 03:52:24.852579 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213957 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:25.002902 7f29149e4700  1 mon.magna102@0(leader).log v18219 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18219)
>2018-07-11 03:52:34.260056 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213960 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:34.445439 7f29149e4700  1 mon.magna102@0(leader).log v18220 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18220)
>2018-07-11 03:52:42.263439 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213966 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:42.421319 7f29149e4700  1 mon.magna102@0(leader).log v18221 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18221)
>2018-07-11 03:52:43.913224 7f29149e4700  1 mon.magna102@0(leader).log v18222 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18222)
>2018-07-11 03:52:48.266010 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213969 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:48.438915 7f29149e4700  1 mon.magna102@0(leader).log v18223 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18223)
>2018-07-11 03:52:58.270284 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213972 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:52:58.439785 7f29149e4700  1 mon.magna102@0(leader).log v18224 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18224)
>2018-07-11 03:53:00.233054 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:53:07.315061 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213975 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:07.440478 7f29149e4700  1 mon.magna102@0(leader).log v18225 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18225)
>2018-07-11 03:53:12.315302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213978 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:12.434045 7f29149e4700  1 mon.magna102@0(leader).log v18226 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18226)
>2018-07-11 03:53:17.315532 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213981 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:17.409596 7f29149e4700  1 mon.magna102@0(leader).log v18227 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18227)
>2018-07-11 03:53:27.315933 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213984 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:27.418833 7f29149e4700  1 mon.magna102@0(leader).log v18228 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18228)
>2018-07-11 03:53:32.319121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213987 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:32.436165 7f29149e4700  1 mon.magna102@0(leader).log v18229 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18229)
>2018-07-11 03:53:37.319333 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213990 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:37.419944 7f29149e4700  1 mon.magna102@0(leader).log v18230 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18230)
>2018-07-11 03:53:47.320392 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213993 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:47.429175 7f29149e4700  1 mon.magna102@0(leader).log v18231 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18231)
>2018-07-11 03:53:52.329455 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213996 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:52.446505 7f29149e4700  1 mon.magna102@0(leader).log v18232 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18232)
>2018-07-11 03:53:57.329663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24213999 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:53:57.430307 7f29149e4700  1 mon.magna102@0(leader).log v18233 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18233)
>2018-07-11 03:54:00.233246 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:54:02.330575 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214002 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:02.447573 7f29149e4700  1 mon.magna102@0(leader).log v18234 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18234)
>2018-07-11 03:54:07.330788 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214005 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:07.432637 7f29149e4700  1 mon.magna102@0(leader).log v18235 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18235)
>2018-07-11 03:54:16.303460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214011 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:16.466749 7f29149e4700  1 mon.magna102@0(leader).log v18236 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18236)
>2018-07-11 03:54:24.306954 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214014 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:24.467491 7f29149e4700  1 mon.magna102@0(leader).log v18237 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18237)
>2018-07-11 03:54:32.351306 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214017 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:32.484997 7f29149e4700  1 mon.magna102@0(leader).log v18238 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18238)
>2018-07-11 03:54:38.312996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214020 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:38.485610 7f29149e4700  1 mon.magna102@0(leader).log v18239 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18239)
>2018-07-11 03:54:46.316398 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214026 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:46.469622 7f29149e4700  1 mon.magna102@0(leader).log v18240 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18240)
>2018-07-11 03:54:58.321524 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214029 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:54:58.470769 7f29149e4700  1 mon.magna102@0(leader).log v18241 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18241)
>2018-07-11 03:55:00.233419 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:55:04.324190 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214032 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:04.479787 7f29149e4700  1 mon.magna102@0(leader).log v18242 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18242)
>2018-07-11 03:55:07.423297 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10115. Immutable memtables: 0.
>
>2018-07-11 03:55:07.423344 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 659] Syncing log #10112
>2018-07-11 03:55:07.439732 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.423334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:55:07.439744 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 659] Flushing memtable with next log file: 10115
>
>2018-07-11 03:55:07.439770 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307439762, "job": 659, "event": "flush_started", "num_memtables": 1, "num_entries": 2037, "num_deletes": 250, "memory_usage": 3890912}
>2018-07-11 03:55:07.439773 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 659] Level-0 flush table #10116: started
>2018-07-11 03:55:07.498608 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307498593, "cf_name": "default", "job": 659, "event": "table_file_creation", "file_number": 10116, "file_size": 2525073, "table_properties": {"data_size": 2505320, "index_size": 10321, "filter_size": 8508, "raw_key_size": 13241, "raw_average_key_size": 20, "raw_value_size": 2489166, "raw_average_value_size": 3944, "num_data_blocks": 317, "num_entries": 631, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 03:55:07.498618 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 659] Level-0 flush table #10116: 2525073 bytes OK
>2018-07-11 03:55:07.548624 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.515140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10116 started
>2018-07-11 03:55:07.548632 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.548585) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10116: memtable #1 done
>2018-07-11 03:55:07.548634 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.548600) EVENT_LOG_v1 {"time_micros": 1531281307548594, "job": 659, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:55:07.548649 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.548614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:55:07.548656 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 659] Try to delete WAL files size 3851584, prev total WAL file size 3851584, number of live WAL files 2.
>
>2018-07-11 03:55:07.549669 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:55:07.549732 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.549725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 03:55:07.549739 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 660] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:55:07.549745 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 659 Base level 0, inputs: [10116(2465KB)], [10114(13MB)]
>
>2018-07-11 03:55:07.549758 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307549748, "job": 660, "event": "compaction_started", "files_L0": [10116], "files_L1": [10114], "score": -1, "input_data_size": 17098105}
>2018-07-11 03:55:07.740994 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 660] Generated table #10117: 3486 keys, 16614707 bytes
>2018-07-11 03:55:07.741038 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307741010, "cf_name": "default", "job": 660, "event": "table_file_creation", "file_number": 10117, "file_size": 16614707, "table_properties": {"data_size": 16487784, "index_size": 72301, "filter_size": 53695, "raw_key_size": 70168, "raw_average_key_size": 20, "raw_value_size": 16387458, "raw_average_value_size": 4700, "num_data_blocks": 2117, "num_entries": 3486, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:55:07.774192 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 660] Compacted 1@0 + 1@1 files to L1 => 16614707 bytes
>2018-07-11 03:55:07.807718 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.807652) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.2 rd, 74.0 wr, level 1, files in(1, 1) out(1) MB in(2.4, 13.9) out(15.8), read-write-amplify(13.4) write-amplify(6.6) OK, records in: 3814, records dropped: 328
>
>2018-07-11 03:55:07.807726 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:07.807673) EVENT_LOG_v1 {"time_micros": 1531281307807664, "job": 660, "event": "compaction_finished", "compaction_time_micros": 224410, "output_level": 1, "num_output_files": 1, "total_output_size": 16614707, "num_input_records": 3814, "num_output_records": 3486, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:55:07.808334 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307808330, "job": 660, "event": "table_file_deletion", "file_number": 10116}
>2018-07-11 03:55:07.811030 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281307811027, "job": 660, "event": "table_file_deletion", "file_number": 10114}
>2018-07-11 03:55:07.811147 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:55:12.389740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214035 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:12.458988 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10118. Immutable memtables: 0.
>
>2018-07-11 03:55:12.492356 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.459044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:55:12.492369 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 661] Flushing memtable with next log file: 10118
>
>2018-07-11 03:55:12.492383 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312492375, "job": 661, "event": "flush_started", "num_memtables": 1, "num_entries": 277, "num_deletes": 251, "memory_usage": 64904}
>2018-07-11 03:55:12.492386 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 661] Level-0 flush table #10119: started
>2018-07-11 03:55:12.545376 7f29149e4700  1 mon.magna102@0(leader).log v18243 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18243)
>2018-07-11 03:55:12.567970 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312567957, "cf_name": "default", "job": 661, "event": "table_file_creation", "file_number": 10119, "file_size": 57285, "table_properties": {"data_size": 55308, "index_size": 246, "filter_size": 812, "raw_key_size": 5322, "raw_average_key_size": 20, "raw_value_size": 51613, "raw_average_value_size": 195, "num_data_blocks": 8, "num_entries": 264, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 03:55:12.567980 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 661] Level-0 flush table #10119: 57285 bytes OK
>2018-07-11 03:55:12.618047 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.584556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10119 started
>2018-07-11 03:55:12.618055 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.618013) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10119: memtable #1 done
>2018-07-11 03:55:12.618058 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.618028) EVENT_LOG_v1 {"time_micros": 1531281312618021, "job": 661, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:55:12.618060 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.618039) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:55:12.618065 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 661] Try to delete WAL files size 56884, prev total WAL file size 74456, number of live WAL files 2.
>
>2018-07-11 03:55:12.618161 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:55:12.618220 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.618215) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 03:55:12.618229 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 662] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:55:12.618235 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 661 Base level 0, inputs: [10119(55KB)], [10117(15MB)]
>
>2018-07-11 03:55:12.618247 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312618238, "job": 662, "event": "compaction_started", "files_L0": [10119], "files_L1": [10117], "score": -1, "input_data_size": 16671992}
>2018-07-11 03:55:12.785391 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 662] Generated table #10120: 3242 keys, 14663195 bytes
>2018-07-11 03:55:12.785423 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312785408, "cf_name": "default", "job": 662, "event": "table_file_creation", "file_number": 10120, "file_size": 14663195, "table_properties": {"data_size": 14550269, "index_size": 64284, "filter_size": 47715, "raw_key_size": 65294, "raw_average_key_size": 20, "raw_value_size": 14458965, "raw_average_value_size": 4459, "num_data_blocks": 1873, "num_entries": 3242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:55:12.818616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 662] Compacted 1@0 + 1@1 files to L1 => 14663195 bytes
>2018-07-11 03:55:12.852138 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.852068) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 83.2 rd, 73.2 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.8) out(14.0), read-write-amplify(547.0) write-amplify(256.0) OK, records in: 3750, records dropped: 508
>
>2018-07-11 03:55:12.852146 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:55:12.852090) EVENT_LOG_v1 {"time_micros": 1531281312852081, "job": 662, "event": "compaction_finished", "compaction_time_micros": 200355, "output_level": 1, "num_output_files": 1, "total_output_size": 14663195, "num_input_records": 3750, "num_output_records": 3242, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:55:12.852241 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312852237, "job": 662, "event": "table_file_deletion", "file_number": 10119}
>2018-07-11 03:55:12.855386 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281312855383, "job": 662, "event": "table_file_deletion", "file_number": 10117}
>2018-07-11 03:55:17.389987 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214038 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:17.469690 7f29149e4700  1 mon.magna102@0(leader).log v18244 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18244)
>2018-07-11 03:55:22.390258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214041 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:22.512039 7f29149e4700  1 mon.magna102@0(leader).log v18245 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18245)
>2018-07-11 03:55:27.390482 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214044 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:27.470832 7f29149e4700  1 mon.magna102@0(leader).log v18246 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18246)
>2018-07-11 03:55:34.337393 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214047 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:34.506256 7f29149e4700  1 mon.magna102@0(leader).log v18247 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18247)
>2018-07-11 03:55:42.340906 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214050 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:42.490344 7f29149e4700  1 mon.magna102@0(leader).log v18248 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18248)
>2018-07-11 03:55:48.343438 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214053 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:48.516007 7f29149e4700  1 mon.magna102@0(leader).log v18249 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18249)
>2018-07-11 03:55:57.391570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214056 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:55:57.483316 7f29149e4700  1 mon.magna102@0(leader).log v18250 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18250)
>2018-07-11 03:56:00.233597 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 03:56:07.391958 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214059 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:07.484269 7f29149e4700  1 mon.magna102@0(leader).log v18251 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18251)
>2018-07-11 03:56:12.392203 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214062 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:12.510045 7f29149e4700  1 mon.magna102@0(leader).log v18252 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18252)
>2018-07-11 03:56:17.392392 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214065 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:17.485492 7f29149e4700  1 mon.magna102@0(leader).log v18253 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18253)
>2018-07-11 03:56:22.394108 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214068 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:22.511159 7f29149e4700  1 mon.magna102@0(leader).log v18254 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18254)
>2018-07-11 03:56:28.360574 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214071 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:28.511720 7f29149e4700  1 mon.magna102@0(leader).log v18255 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18255)
>2018-07-11 03:56:34.363149 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214074 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:34.513515 7f29149e4700  1 mon.magna102@0(leader).log v18256 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18256)
>2018-07-11 03:56:42.405734 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214077 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:42.522724 7f29149e4700  1 mon.magna102@0(leader).log v18257 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18257)
>2018-07-11 03:56:47.405908 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214080 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:47.506470 7f29149e4700  1 mon.magna102@0(leader).log v18258 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18258)
>2018-07-11 03:56:52.406828 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214083 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:52.532221 7f29149e4700  1 mon.magna102@0(leader).log v18259 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18259)
>2018-07-11 03:56:57.407044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214086 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:56:57.532650 7f29149e4700  1 mon.magna102@0(leader).log v18260 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18260)
>2018-07-11 03:57:00.233781 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 03:57:04.375722 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214089 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:04.550076 7f29149e4700  1 mon.magna102@0(leader).log v18261 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18261)
>2018-07-11 03:57:12.417132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214092 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:12.550852 7f29149e4700  1 mon.magna102@0(leader).log v18262 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18262)
>2018-07-11 03:57:18.381798 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214095 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:18.526454 7f29149e4700  1 mon.magna102@0(leader).log v18263 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18263)
>2018-07-11 03:57:27.418510 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214098 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:27.518926 7f29149e4700  1 mon.magna102@0(leader).log v18264 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18264)
>2018-07-11 03:57:32.420446 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214101 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:32.537497 7f29149e4700  1 mon.magna102@0(leader).log v18265 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18265)
>2018-07-11 03:57:37.420649 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214104 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:37.496740 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10121. Immutable memtables: 0.
>
>2018-07-11 03:57:37.496783 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 663] Syncing log #10118
>2018-07-11 03:57:37.513024 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.496773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:57:37.513035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 663] Flushing memtable with next log file: 10121
>
>2018-07-11 03:57:37.513061 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457513053, "job": 663, "event": "flush_started", "num_memtables": 1, "num_entries": 1193, "num_deletes": 500, "memory_usage": 1534896}
>2018-07-11 03:57:37.513064 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 663] Level-0 flush table #10122: started
>2018-07-11 03:57:37.597359 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457597344, "cf_name": "default", "job": 663, "event": "table_file_creation", "file_number": 10122, "file_size": 1471931, "table_properties": {"data_size": 1458740, "index_size": 6376, "filter_size": 5895, "raw_key_size": 14903, "raw_average_key_size": 20, "raw_value_size": 1444363, "raw_average_value_size": 2000, "num_data_blocks": 194, "num_entries": 722, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 03:57:37.597369 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 663] Level-0 flush table #10122: 1471931 bytes OK
>2018-07-11 03:57:37.638880 7f29149e4700  1 mon.magna102@0(leader).log v18266 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18266)
>2018-07-11 03:57:37.680501 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.638651) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10122 started
>2018-07-11 03:57:37.680510 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.680428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10122: memtable #1 done
>2018-07-11 03:57:37.680513 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.680443) EVENT_LOG_v1 {"time_micros": 1531281457680437, "job": 663, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:57:37.680515 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.680455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:57:37.680524 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 663] Try to delete WAL files size 1508232, prev total WAL file size 1526644, number of live WAL files 2.
>
>2018-07-11 03:57:37.680958 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:57:37.681135 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 03:57:37.681142 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 43484.5 total, 650.3 interval
>Cumulative writes: 61K writes, 294K keys, 61K commit groups, 1.0 writes per commit group, ingest: 0.50 GB, 0.01 MB/s
>Cumulative WAL: 61K writes, 61K syncs, 1.00 writes per sync, written: 0.50 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 862 writes, 4363 keys, 862 commit groups, 1.0 writes per commit group, ingest: 6.41 MB, 0.01 MB/s
>Interval WAL: 862 writes, 862 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.40 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        28       332    0.083       0      0
>  L1      1/1   13.98 MB   0.0      5.2     0.4      4.8       4.8     -0.0       0.0  10.9     72.4     66.3        74       330    0.223   1166K   137K
> Sum      2/2   15.39 MB   0.0      5.2     0.4      4.8       5.2      0.4       0.0  11.9     52.6     52.6       101       662    0.153   1166K   137K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  12.5     51.1     51.0         1         8    0.155     14K   1737
>Uptime(secs): 43484.5 total, 43484.5 interval
>Flush(GB): cumulative 0.439, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.20 GB write, 0.12 MB/s write, 5.20 GB read, 0.12 MB/s read, 101.3 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.40 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        28       332    0.083       0      0
>  L1      1/1   13.98 MB   0.0      5.2     0.4      4.8       4.8     -0.0       0.0  10.9     72.4     66.3        74       330    0.223   1166K   137K
> Sum      2/2   15.39 MB   0.0      5.2     0.4      4.8       5.2      0.4       0.0  11.9     52.6     52.6       101       662    0.153   1166K   137K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 43484.5 total, 43484.5 interval
>Flush(GB): cumulative 0.439, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.20 GB write, 0.12 MB/s write, 5.20 GB read, 0.12 MB/s read, 101.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 03:57:37.681162 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.681155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 03:57:37.681167 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 664] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:57:37.681172 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 663 Base level 0, inputs: [10122(1437KB)], [10120(13MB)]
>
>2018-07-11 03:57:37.681208 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457681175, "job": 664, "event": "compaction_started", "files_L0": [10122], "files_L1": [10120], "score": -1, "input_data_size": 16135126}
>2018-07-11 03:57:37.856212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 664] Generated table #10123: 2956 keys, 13809094 bytes
>2018-07-11 03:57:37.856260 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457856233, "cf_name": "default", "job": 664, "event": "table_file_creation", "file_number": 10123, "file_size": 13809094, "table_properties": {"data_size": 13701741, "index_size": 61474, "filter_size": 44952, "raw_key_size": 59491, "raw_average_key_size": 20, "raw_value_size": 13616491, "raw_average_value_size": 4606, "num_data_blocks": 1803, "num_entries": 2956, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:57:37.897719 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 664] Compacted 1@0 + 1@1 files to L1 => 13809094 bytes
>2018-07-11 03:57:37.939579 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.939508) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.5 rd, 63.8 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(13.2), read-write-amplify(20.3) write-amplify(9.4) OK, records in: 3964, records dropped: 1008
>
>2018-07-11 03:57:37.939588 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.939529) EVENT_LOG_v1 {"time_micros": 1531281457939520, "job": 664, "event": "compaction_finished", "compaction_time_micros": 216494, "output_level": 1, "num_output_files": 1, "total_output_size": 13809094, "num_input_records": 3964, "num_output_records": 2956, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:57:37.939941 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457939937, "job": 664, "event": "table_file_deletion", "file_number": 10122}
>2018-07-11 03:57:37.942746 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457942743, "job": 664, "event": "table_file_deletion", "file_number": 10120}
>2018-07-11 03:57:37.942997 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10124. Immutable memtables: 0.
>
>2018-07-11 03:57:37.943037 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 665] Syncing log #10121
>2018-07-11 03:57:37.956548 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:37.943029) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 03:57:37.956560 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 665] Flushing memtable with next log file: 10124
>
>2018-07-11 03:57:37.956572 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457956566, "job": 665, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20768}
>2018-07-11 03:57:37.956575 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 665] Level-0 flush table #10125: started
>2018-07-11 03:57:37.998698 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281457998685, "cf_name": "default", "job": 665, "event": "table_file_creation", "file_number": 10125, "file_size": 19527, "table_properties": {"data_size": 18461, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18238, "raw_average_value_size": 2279, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:57:37.998708 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 665] Level-0 flush table #10125: 19527 bytes OK
>2018-07-11 03:57:38.048733 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.015256) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10125 started
>2018-07-11 03:57:38.048741 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.048699) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10125: memtable #1 done
>2018-07-11 03:57:38.048744 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.048714) EVENT_LOG_v1 {"time_micros": 1531281458048708, "job": 665, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 03:57:38.048746 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.048725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 03:57:38.048751 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 665] Try to delete WAL files size 18412, prev total WAL file size 18412, number of live WAL files 2.
>
>2018-07-11 03:57:38.048843 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 03:57:38.048894 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.048888) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 03:57:38.048903 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 666] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 03:57:38.048908 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 665 Base level 0, inputs: [10125(19KB)], [10123(13MB)]
>
>2018-07-11 03:57:38.048921 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281458048911, "job": 666, "event": "compaction_started", "files_L0": [10125], "files_L1": [10123], "score": -1, "input_data_size": 13828621}
>2018-07-11 03:57:38.216146 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 666] Generated table #10126: 2959 keys, 13827507 bytes
>2018-07-11 03:57:38.216208 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281458216167, "cf_name": "default", "job": 666, "event": "table_file_creation", "file_number": 10126, "file_size": 13827507, "table_properties": {"data_size": 13720029, "index_size": 61543, "filter_size": 45008, "raw_key_size": 59552, "raw_average_key_size": 20, "raw_value_size": 13634689, "raw_average_value_size": 4607, "num_data_blocks": 1805, "num_entries": 2959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 03:57:38.249345 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 666] Compacted 1@0 + 1@1 files to L1 => 13827507 bytes
>2018-07-11 03:57:38.282876 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.282805) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.0 rd, 69.0 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.2) out(13.2), read-write-amplify(1416.3) write-amplify(708.1) OK, records in: 2964, records dropped: 5
>
>2018-07-11 03:57:38.282885 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-03:57:38.282827) EVENT_LOG_v1 {"time_micros": 1531281458282817, "job": 666, "event": "compaction_finished", "compaction_time_micros": 200408, "output_level": 1, "num_output_files": 1, "total_output_size": 13827507, "num_input_records": 2964, "num_output_records": 2959, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 03:57:38.282962 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281458282958, "job": 666, "event": "table_file_deletion", "file_number": 10125}
>2018-07-11 03:57:38.285609 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281458285604, "job": 666, "event": "table_file_deletion", "file_number": 10123}
>2018-07-11 03:57:42.463269 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214107 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:43.525638 7f29149e4700  1 mon.magna102@0(leader).log v18267 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18267)
>2018-07-11 03:57:48.395027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214110 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:48.551258 7f29149e4700  1 mon.magna102@0(leader).log v18268 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18268)
>2018-07-11 03:57:54.397417 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214113 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:57:54.551847 7f29149e4700  1 mon.magna102@0(leader).log v18269 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18269)
>2018-07-11 03:58:00.233970 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:58:02.464063 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214116 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:02.595622 7f29149e4700  1 mon.magna102@0(leader).log v18270 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18270)
>2018-07-11 03:58:08.403460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214119 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:08.554546 7f29149e4700  1 mon.magna102@0(leader).log v18271 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18271)
>2018-07-11 03:58:14.406146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214122 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:14.555131 7f29149e4700  1 mon.magna102@0(leader).log v18272 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18272)
>2018-07-11 03:58:22.409500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214128 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:22.556403 7f29149e4700  1 mon.magna102@0(leader).log v18273 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18273)
>2018-07-11 03:58:34.414626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214131 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:34.599073 7f29149e4700  1 mon.magna102@0(leader).log v18274 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18274)
>2018-07-11 03:58:42.418124 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214137 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:42.566561 7f29149e4700  1 mon.magna102@0(leader).log v18275 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18275)
>2018-07-11 03:58:48.420706 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214140 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:48.575510 7f29149e4700  1 mon.magna102@0(leader).log v18276 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18276)
>2018-07-11 03:58:58.425009 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214143 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:58:58.586160 7f29149e4700  1 mon.magna102@0(leader).log v18277 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18277)
>2018-07-11 03:59:00.234153 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 03:59:07.466337 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214146 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:07.570216 7f29149e4700  1 mon.magna102@0(leader).log v18278 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18278)
>2018-07-11 03:59:12.470503 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214149 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:12.587547 7f29149e4700  1 mon.magna102@0(leader).log v18279 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18279)
>2018-07-11 03:59:17.470706 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214152 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:17.571334 7f29149e4700  1 mon.magna102@0(leader).log v18280 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18280)
>2018-07-11 03:59:22.471629 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214155 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:22.596986 7f29149e4700  1 mon.magna102@0(leader).log v18281 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18281)
>2018-07-11 03:59:28.437895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214158 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:28.605928 7f29149e4700  1 mon.magna102@0(leader).log v18282 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18282)
>2018-07-11 03:59:34.440252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214161 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:34.614856 7f29149e4700  1 mon.magna102@0(leader).log v18283 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18283)
>2018-07-11 03:59:42.481950 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214164 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:42.615615 7f29149e4700  1 mon.magna102@0(leader).log v18284 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18284)
>2018-07-11 03:59:48.446198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214167 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:48.591332 7f29149e4700  1 mon.magna102@0(leader).log v18285 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18285)
>2018-07-11 03:59:57.484521 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214170 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 03:59:57.584955 7f29149e4700  1 mon.magna102@0(leader).log v18286 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18286)
>2018-07-11 04:00:00.234350 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:00:02.494015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214173 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:02.611082 7f29149e4700  1 mon.magna102@0(leader).log v18287 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18287)
>2018-07-11 04:00:07.494262 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214176 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:07.594855 7f29149e4700  1 mon.magna102@0(leader).log v18288 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18288)
>2018-07-11 04:00:12.495185 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214179 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:12.612213 7f29149e4700  1 mon.magna102@0(leader).log v18289 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18289)
>2018-07-11 04:00:18.459119 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214182 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:18.612796 7f29149e4700  1 mon.magna102@0(leader).log v18290 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18290)
>2018-07-11 04:00:26.462547 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214188 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:26.613521 7f29149e4700  1 mon.magna102@0(leader).log v18291 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18291)
>2018-07-11 04:00:38.467516 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214191 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:38.622993 7f29149e4700  1 mon.magna102@0(leader).log v18292 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18292)
>2018-07-11 04:00:44.470146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214194 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:44.623678 7f29149e4700  1 mon.magna102@0(leader).log v18293 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18293)
>2018-07-11 04:00:52.516980 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214197 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:52.650657 7f29149e4700  1 mon.magna102@0(leader).log v18294 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18294)
>2018-07-11 04:00:57.517208 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214200 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:00:57.642777 7f29149e4700  1 mon.magna102@0(leader).log v18295 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18295)
>2018-07-11 04:01:00.234526 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:01:02.518852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214203 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:02.644223 7f29149e4700  1 mon.magna102@0(leader).log v18296 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18296)
>2018-07-11 04:01:07.519094 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214206 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:07.611333 7f29149e4700  1 mon.magna102@0(leader).log v18297 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18297)
>2018-07-11 04:01:14.483144 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214209 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:14.646536 7f29149e4700  1 mon.magna102@0(leader).log v18298 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18298)
>2018-07-11 04:01:22.486513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214212 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:22.637916 7f29149e4700  1 mon.magna102@0(leader).log v18299 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18299)
>2018-07-11 04:01:27.521098 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214215 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:27.621694 7f29149e4700  1 mon.magna102@0(leader).log v18300 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18300)
>2018-07-11 04:01:34.491690 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214218 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:34.622725 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10127. Immutable memtables: 0.
>
>2018-07-11 04:01:34.622770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 667] Syncing log #10124
>2018-07-11 04:01:34.639219 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.622760) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:01:34.639229 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 667] Flushing memtable with next log file: 10127
>
>2018-07-11 04:01:34.639256 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281694639248, "job": 667, "event": "flush_started", "num_memtables": 1, "num_entries": 1361, "num_deletes": 251, "memory_usage": 2444016}
>2018-07-11 04:01:34.639259 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 667] Level-0 flush table #10128: started
>2018-07-11 04:01:34.791230 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281694791214, "cf_name": "default", "job": 667, "event": "table_file_creation", "file_number": 10128, "file_size": 2354440, "table_properties": {"data_size": 2335260, "index_size": 10204, "filter_size": 8052, "raw_key_size": 12319, "raw_average_key_size": 20, "raw_value_size": 2319589, "raw_average_value_size": 3865, "num_data_blocks": 309, "num_entries": 600, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:01:34.791240 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 667] Level-0 flush table #10128: 2354440 bytes OK
>2018-07-11 04:01:34.824539 7f29149e4700  1 mon.magna102@0(leader).log v18301 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18301)
>2018-07-11 04:01:34.857907 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.824290) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10128 started
>2018-07-11 04:01:34.857915 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.857870) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10128: memtable #1 done
>2018-07-11 04:01:34.857918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.857885) EVENT_LOG_v1 {"time_micros": 1531281694857878, "job": 667, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:01:34.857920 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.857897) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:01:34.857926 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 667] Try to delete WAL files size 2416104, prev total WAL file size 2434804, number of live WAL files 2.
>
>2018-07-11 04:01:34.858603 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:01:34.858657 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:01:34.858650) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:01:34.858666 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 668] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:01:34.858672 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 667 Base level 0, inputs: [10128(2299KB)], [10126(13MB)]
>
>2018-07-11 04:01:34.858685 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281694858675, "job": 668, "event": "compaction_started", "files_L0": [10128], "files_L1": [10126], "score": -1, "input_data_size": 16181947}
>2018-07-11 04:01:35.033607 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 668] Generated table #10129: 3049 keys, 14189903 bytes
>2018-07-11 04:01:35.033655 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281695033627, "cf_name": "default", "job": 668, "event": "table_file_creation", "file_number": 10129, "file_size": 14189903, "table_properties": {"data_size": 14078927, "index_size": 63756, "filter_size": 46293, "raw_key_size": 61624, "raw_average_key_size": 20, "raw_value_size": 13990680, "raw_average_value_size": 4588, "num_data_blocks": 1862, "num_entries": 3049, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:01:35.066722 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 668] Compacted 1@0 + 1@1 files to L1 => 14189903 bytes
>2018-07-11 04:01:35.100326 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:01:35.100257) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.8 rd, 68.2 wr, level 1, files in(1, 1) out(1) MB in(2.2, 13.2) out(13.5), read-write-amplify(12.9) write-amplify(6.0) OK, records in: 3559, records dropped: 510
>
>2018-07-11 04:01:35.100335 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:01:35.100278) EVENT_LOG_v1 {"time_micros": 1531281695100269, "job": 668, "event": "compaction_finished", "compaction_time_micros": 208021, "output_level": 1, "num_output_files": 1, "total_output_size": 14189903, "num_input_records": 3559, "num_output_records": 3049, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:01:35.100847 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281695100843, "job": 668, "event": "table_file_deletion", "file_number": 10128}
>2018-07-11 04:01:35.103480 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281695103477, "job": 668, "event": "table_file_deletion", "file_number": 10126}
>2018-07-11 04:01:40.494275 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214221 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:40.668012 7f29149e4700  1 mon.magna102@0(leader).log v18302 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18302)
>2018-07-11 04:01:46.497023 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214224 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:46.676963 7f29149e4700  1 mon.magna102@0(leader).log v18303 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18303)
>2018-07-11 04:01:52.535074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214227 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:52.652457 7f29149e4700  1 mon.magna102@0(leader).log v18304 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18304)
>2018-07-11 04:01:58.502072 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214230 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:01:58.653208 7f29149e4700  1 mon.magna102@0(leader).log v18305 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18305)
>2018-07-11 04:02:00.234712 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:02:07.536835 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214233 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:07.628995 7f29149e4700  1 mon.magna102@0(leader).log v18306 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18306)
>2018-07-11 04:02:14.508997 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214236 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:14.663104 7f29149e4700  1 mon.magna102@0(leader).log v18307 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18307)
>2018-07-11 04:02:20.511498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214239 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:20.664934 7f29149e4700  1 mon.magna102@0(leader).log v18308 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18308)
>2018-07-11 04:02:27.548341 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214242 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:27.657178 7f29149e4700  1 mon.magna102@0(leader).log v18309 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18309)
>2018-07-11 04:02:32.549152 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214245 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:32.666207 7f29149e4700  1 mon.magna102@0(leader).log v18310 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18310)
>2018-07-11 04:02:38.519198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214248 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:38.675125 7f29149e4700  1 mon.magna102@0(leader).log v18311 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18311)
>2018-07-11 04:02:44.521738 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214251 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:44.675788 7f29149e4700  1 mon.magna102@0(leader).log v18312 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18312)
>2018-07-11 04:02:52.559513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214254 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:52.676554 7f29149e4700  1 mon.magna102@0(leader).log v18313 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18313)
>2018-07-11 04:02:57.559724 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214257 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:02:57.693665 7f29149e4700  1 mon.magna102@0(leader).log v18314 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18314)
>2018-07-11 04:03:00.234902 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:03:02.569076 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214260 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:02.702688 7f29149e4700  1 mon.magna102@0(leader).log v18315 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18315)
>2018-07-11 04:03:07.569283 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214263 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:07.694880 7f29149e4700  1 mon.magna102@0(leader).log v18316 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18316)
>2018-07-11 04:03:14.534466 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214266 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:14.737328 7f29149e4700  1 mon.magna102@0(leader).log v18317 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18317)
>2018-07-11 04:03:22.572267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214269 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:22.689318 7f29149e4700  1 mon.magna102@0(leader).log v18318 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18318)
>2018-07-11 04:03:27.639945 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10130. Immutable memtables: 0.
>
>2018-07-11 04:03:27.639988 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 669] Syncing log #10127
>2018-07-11 04:03:27.656502 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.639979) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:03:27.656513 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 669] Flushing memtable with next log file: 10130
>
>2018-07-11 04:03:27.656539 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807656531, "job": 669, "event": "flush_started", "num_memtables": 1, "num_entries": 791, "num_deletes": 250, "memory_usage": 1199600}
>2018-07-11 04:03:27.656542 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 669] Level-0 flush table #10131: started
>2018-07-11 04:03:27.706950 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807706936, "cf_name": "default", "job": 669, "event": "table_file_creation", "file_number": 10131, "file_size": 812697, "table_properties": {"data_size": 805363, "index_size": 3260, "filter_size": 3155, "raw_key_size": 8079, "raw_average_key_size": 21, "raw_value_size": 797872, "raw_average_value_size": 2127, "num_data_blocks": 101, "num_entries": 375, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:03:27.706961 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 669] Level-0 flush table #10131: 812697 bytes OK
>2018-07-11 04:03:27.765396 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.723538) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10131 started
>2018-07-11 04:03:27.765404 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.765360) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10131: memtable #1 done
>2018-07-11 04:03:27.765406 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.765375) EVENT_LOG_v1 {"time_micros": 1531281807765368, "job": 669, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:03:27.765421 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.765386) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:03:27.765427 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 669] Try to delete WAL files size 1182467, prev total WAL file size 1182467, number of live WAL files 2.
>
>2018-07-11 04:03:27.765841 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:03:27.765904 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.765898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:03:27.765911 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 670] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:03:27.765917 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 669 Base level 0, inputs: [10131(793KB)], [10129(13MB)]
>
>2018-07-11 04:03:27.765931 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807765921, "job": 670, "event": "compaction_started", "files_L0": [10131], "files_L1": [10129], "score": -1, "input_data_size": 15002600}
>2018-07-11 04:03:27.924414 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 670] Generated table #10132: 2966 keys, 13631662 bytes
>2018-07-11 04:03:27.924466 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807924450, "cf_name": "default", "job": 670, "event": "table_file_creation", "file_number": 10132, "file_size": 13631662, "table_properties": {"data_size": 13526257, "index_size": 60128, "filter_size": 44350, "raw_key_size": 59594, "raw_average_key_size": 20, "raw_value_size": 13441825, "raw_average_value_size": 4531, "num_data_blocks": 1762, "num_entries": 2966, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:03:27.957623 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 670] Compacted 1@0 + 1@1 files to L1 => 13631662 bytes
>2018-07-11 04:03:27.991147 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.991078) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 78.3 rd, 71.1 wr, level 1, files in(1, 1) out(1) MB in(0.8, 13.5) out(13.0), read-write-amplify(35.2) write-amplify(16.8) OK, records in: 3424, records dropped: 458
>
>2018-07-11 04:03:27.991155 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:03:27.991100) EVENT_LOG_v1 {"time_micros": 1531281807991091, "job": 670, "event": "compaction_finished", "compaction_time_micros": 191675, "output_level": 1, "num_output_files": 1, "total_output_size": 13631662, "num_input_records": 3424, "num_output_records": 2966, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:03:27.991398 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807991394, "job": 670, "event": "table_file_deletion", "file_number": 10131}
>2018-07-11 04:03:27.994035 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281807994032, "job": 670, "event": "table_file_deletion", "file_number": 10129}
>2018-07-11 04:03:27.994150 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:03:28.639796 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214272 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:28.791939 7f29149e4700  1 mon.magna102@0(leader).log v18319 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18319)
>2018-07-11 04:03:34.543211 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214275 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:34.692646 7f29149e4700  1 mon.magna102@0(leader).log v18320 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18320)
>2018-07-11 04:03:42.606893 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214278 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:42.726662 7f29149e4700  1 mon.magna102@0(leader).log v18321 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18321)
>2018-07-11 04:03:47.607109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214281 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:47.693849 7f29149e4700  1 mon.magna102@0(leader).log v18322 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18322)
>2018-07-11 04:03:57.607521 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214284 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:03:57.694763 7f29149e4700  1 mon.magna102@0(leader).log v18323 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18323)
>2018-07-11 04:04:00.235079 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:04:02.607753 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214287 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:02.738341 7f29149e4700  1 mon.magna102@0(leader).log v18324 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18324)
>2018-07-11 04:04:07.607968 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214290 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:07.730525 7f29149e4700  1 mon.magna102@0(leader).log v18325 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18325)
>2018-07-11 04:04:12.608209 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214293 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:12.739476 7f29149e4700  1 mon.magna102@0(leader).log v18326 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18326)
>2018-07-11 04:04:18.562042 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214296 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:18.731659 7f29149e4700  1 mon.magna102@0(leader).log v18327 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18327)
>2018-07-11 04:04:27.608761 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214299 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:27.707477 7f29149e4700  1 mon.magna102@0(leader).log v18328 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18328)
>2018-07-11 04:04:37.609171 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214302 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:37.700084 7f29149e4700  1 mon.magna102@0(leader).log v18329 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18329)
>2018-07-11 04:04:42.609382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214305 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:42.725747 7f29149e4700  1 mon.magna102@0(leader).log v18330 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18330)
>2018-07-11 04:04:47.609587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214308 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:47.734692 7f29149e4700  1 mon.magna102@0(leader).log v18331 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18331)
>2018-07-11 04:04:52.609988 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214311 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:52.726957 7f29149e4700  1 mon.magna102@0(leader).log v18332 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18332)
>2018-07-11 04:04:57.610198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214314 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:04:57.720387 7f29149e4700  1 mon.magna102@0(leader).log v18333 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18333)
>2018-07-11 04:05:00.235259 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:05:06.582617 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214320 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:06.737935 7f29149e4700  1 mon.magna102@0(leader).log v18334 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18334)
>2018-07-11 04:05:12.585251 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214323 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:12.738475 7f29149e4700  1 mon.magna102@0(leader).log v18335 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18335)
>2018-07-11 04:05:20.588499 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214326 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:20.739283 7f29149e4700  1 mon.magna102@0(leader).log v18336 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18336)
>2018-07-11 04:05:27.647784 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214329 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:27.731521 7f29149e4700  1 mon.magna102@0(leader).log v18337 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18337)
>2018-07-11 04:05:34.594480 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214332 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:34.765648 7f29149e4700  1 mon.magna102@0(leader).log v18338 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18338)
>2018-07-11 04:05:42.648371 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214335 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:42.783088 7f29149e4700  1 mon.magna102@0(leader).log v18339 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18339)
>2018-07-11 04:05:47.648613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214338 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:47.758628 7f29149e4700  1 mon.magna102@0(leader).log v18340 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18340)
>2018-07-11 04:05:52.648840 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214341 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:05:52.784302 7f29149e4700  1 mon.magna102@0(leader).log v18341 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18341)
>2018-07-11 04:06:00.235451 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:06:00.605360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214344 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:00.752922 7f29149e4700  1 mon.magna102@0(leader).log v18342 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18342)
>2018-07-11 04:06:07.649404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214347 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:07.745176 7f29149e4700  1 mon.magna102@0(leader).log v18343 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18343)
>2018-07-11 04:06:12.649639 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214350 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:12.770766 7f29149e4700  1 mon.magna102@0(leader).log v18344 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18344)
>2018-07-11 04:06:17.649859 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214353 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:17.746304 7f29149e4700  1 mon.magna102@0(leader).log v18345 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18345)
>2018-07-11 04:06:24.615652 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214356 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:24.763714 7f29149e4700  1 mon.magna102@0(leader).log v18346 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18346)
>2018-07-11 04:06:32.655804 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214359 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:32.772820 7f29149e4700  1 mon.magna102@0(leader).log v18347 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18347)
>2018-07-11 04:06:37.656005 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214362 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:37.756684 7f29149e4700  1 mon.magna102@0(leader).log v18348 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18348)
>2018-07-11 04:06:47.657142 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214365 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:47.757565 7f29149e4700  1 mon.magna102@0(leader).log v18349 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18349)
>2018-07-11 04:06:52.667410 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214368 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:52.784519 7f29149e4700  1 mon.magna102@0(leader).log v18350 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18350)
>2018-07-11 04:06:57.667637 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214371 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:06:57.785057 7f29149e4700  1 mon.magna102@0(leader).log v18351 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18351)
>2018-07-11 04:07:00.235636 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:07:02.668680 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214374 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:02.802386 7f29149e4700  1 mon.magna102@0(leader).log v18352 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18352)
>2018-07-11 04:07:08.634422 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214377 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:08.803094 7f29149e4700  1 mon.magna102@0(leader).log v18353 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18353)
>2018-07-11 04:07:14.637074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214380 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:14.812018 7f29149e4700  1 mon.magna102@0(leader).log v18354 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18354)
>2018-07-11 04:07:22.678522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214383 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:22.796026 7f29149e4700  1 mon.magna102@0(leader).log v18355 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18355)
>2018-07-11 04:07:27.678734 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214386 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:27.771476 7f29149e4700  1 mon.magna102@0(leader).log v18356 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18356)
>2018-07-11 04:07:32.680231 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214389 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:32.797134 7f29149e4700  1 mon.magna102@0(leader).log v18357 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18357)
>2018-07-11 04:07:36.580741 7f29149e4700  1 mon.magna102@0(leader).log v18358 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18358)
>2018-07-11 04:07:38.647637 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214392 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:38.797992 7f29149e4700  1 mon.magna102@0(leader).log v18359 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18359)
>2018-07-11 04:07:39.864777 7f29149e4700  1 mon.magna102@0(leader).log v18360 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18360)
>2018-07-11 04:07:47.681848 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214395 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:47.791830 7f29149e4700  1 mon.magna102@0(leader).log v18361 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18361)
>2018-07-11 04:07:50.792547 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10133. Immutable memtables: 0.
>
>2018-07-11 04:07:50.792590 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 671] Syncing log #10130
>2018-07-11 04:07:50.809085 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.792580) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:07:50.809097 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 671] Flushing memtable with next log file: 10133
>
>2018-07-11 04:07:50.809123 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282070809115, "job": 671, "event": "flush_started", "num_memtables": 1, "num_entries": 1524, "num_deletes": 251, "memory_usage": 2809112}
>2018-07-11 04:07:50.809126 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 671] Level-0 flush table #10134: started
>2018-07-11 04:07:50.867902 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282070867887, "cf_name": "default", "job": 671, "event": "table_file_creation", "file_number": 10134, "file_size": 2709384, "table_properties": {"data_size": 2687569, "index_size": 11732, "filter_size": 9159, "raw_key_size": 13414, "raw_average_key_size": 20, "raw_value_size": 2670072, "raw_average_value_size": 4088, "num_data_blocks": 354, "num_entries": 653, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:07:50.867926 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 671] Level-0 flush table #10134: 2709384 bytes OK
>2018-07-11 04:07:50.917917 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.884414) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10134 started
>2018-07-11 04:07:50.917925 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.917877) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10134: memtable #1 done
>2018-07-11 04:07:50.917928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.917892) EVENT_LOG_v1 {"time_micros": 1531282070917886, "job": 671, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:07:50.917930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.917906) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:07:50.917937 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 671] Try to delete WAL files size 2777923, prev total WAL file size 2777923, number of live WAL files 2.
>
>2018-07-11 04:07:50.918665 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:07:50.918844 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 04:07:50.918851 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 44097.8 total, 613.2 interval
>Cumulative writes: 62K writes, 298K keys, 62K commit groups, 1.0 writes per commit group, ingest: 0.50 GB, 0.01 MB/s
>Cumulative WAL: 62K writes, 62K syncs, 1.00 writes per sync, written: 0.50 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 810 writes, 3676 keys, 810 commit groups, 1.0 writes per commit group, ingest: 6.08 MB, 0.01 MB/s
>Interval WAL: 810 writes, 810 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.58 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.2        28       336    0.083       0      0
>  L1      1/1   13.00 MB   0.0      5.3     0.4      4.8       4.8     -0.0       0.0  10.9     72.3     66.2        74       334    0.223   1180K   139K
> Sum      2/2   15.58 MB   0.0      5.3     0.4      4.8       5.3      0.4       0.0  11.8     52.6     52.6       102       670    0.153   1180K   139K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.4     48.5     48.6         1         8    0.150     13K   1981
>Uptime(secs): 44097.8 total, 44097.8 interval
>Flush(GB): cumulative 0.444, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.26 GB write, 0.12 MB/s write, 5.26 GB read, 0.12 MB/s read, 102.5 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.58 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.2        28       336    0.083       0      0
>  L1      1/1   13.00 MB   0.0      5.3     0.4      4.8       4.8     -0.0       0.0  10.9     72.3     66.2        74       334    0.223   1180K   139K
> Sum      2/2   15.58 MB   0.0      5.3     0.4      4.8       5.3      0.4       0.0  11.8     52.6     52.6       102       670    0.153   1180K   139K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 44097.8 total, 44097.8 interval
>Flush(GB): cumulative 0.444, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.26 GB write, 0.12 MB/s write, 5.26 GB read, 0.12 MB/s read, 102.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 04:07:50.918872 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:07:50.918866) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:07:50.918877 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 672] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:07:50.918882 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 671 Base level 0, inputs: [10134(2645KB)], [10132(13MB)]
>
>2018-07-11 04:07:50.918894 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282070918885, "job": 672, "event": "compaction_started", "files_L0": [10134], "files_L1": [10132], "score": -1, "input_data_size": 16341046}
>2018-07-11 04:07:51.085359 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 672] Generated table #10135: 3109 keys, 14357586 bytes
>2018-07-11 04:07:51.085405 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282071085389, "cf_name": "default", "job": 672, "event": "table_file_creation", "file_number": 10135, "file_size": 14357586, "table_properties": {"data_size": 14246026, "index_size": 63870, "filter_size": 46763, "raw_key_size": 62761, "raw_average_key_size": 20, "raw_value_size": 14156859, "raw_average_value_size": 4553, "num_data_blocks": 1864, "num_entries": 3109, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:07:51.118555 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 672] Compacted 1@0 + 1@1 files to L1 => 14357586 bytes
>2018-07-11 04:07:51.152090 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:07:51.152024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 81.9 rd, 71.9 wr, level 1, files in(1, 1) out(1) MB in(2.6, 13.0) out(13.7), read-write-amplify(11.3) write-amplify(5.3) OK, records in: 3619, records dropped: 510
>
>2018-07-11 04:07:51.152099 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:07:51.152046) EVENT_LOG_v1 {"time_micros": 1531282071152036, "job": 672, "event": "compaction_finished", "compaction_time_micros": 199643, "output_level": 1, "num_output_files": 1, "total_output_size": 14357586, "num_input_records": 3619, "num_output_records": 3109, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:07:51.152732 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282071152728, "job": 672, "event": "table_file_deletion", "file_number": 10134}
>2018-07-11 04:07:51.155413 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282071155411, "job": 672, "event": "table_file_deletion", "file_number": 10132}
>2018-07-11 04:07:52.702365 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214398 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:52.819509 7f29149e4700  1 mon.magna102@0(leader).log v18362 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18362)
>2018-07-11 04:07:57.702561 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214401 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:07:57.786625 7f29149e4700  1 mon.magna102@0(leader).log v18363 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18363)
>2018-07-11 04:08:00.235817 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:08:07.702918 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214404 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:07.795978 7f29149e4700  1 mon.magna102@0(leader).log v18364 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18364)
>2018-07-11 04:08:12.703113 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214407 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:12.821628 7f29149e4700  1 mon.magna102@0(leader).log v18365 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18365)
>2018-07-11 04:08:18.664498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214410 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:18.813895 7f29149e4700  1 mon.magna102@0(leader).log v18366 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18366)
>2018-07-11 04:08:24.667084 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214413 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:24.823008 7f29149e4700  1 mon.magna102@0(leader).log v18367 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18367)
>2018-07-11 04:08:32.670593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214416 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:32.841885 7f29149e4700  1 mon.magna102@0(leader).log v18368 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18368)
>2018-07-11 04:08:37.708294 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214419 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:37.833969 7f29149e4700  1 mon.magna102@0(leader).log v18369 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18369)
>2018-07-11 04:08:47.717777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214422 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:47.809978 7f29149e4700  1 mon.magna102@0(leader).log v18370 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18370)
>2018-07-11 04:08:52.718006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214425 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:52.835769 7f29149e4700  1 mon.magna102@0(leader).log v18371 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18371)
>2018-07-11 04:08:57.718232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214428 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:08:57.827983 7f29149e4700  1 mon.magna102@0(leader).log v18372 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18372)
>2018-07-11 04:09:00.236004 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:09:02.718437 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214431 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:02.836985 7f29149e4700  1 mon.magna102@0(leader).log v18373 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18373)
>2018-07-11 04:09:07.718646 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214434 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:07.829104 7f29149e4700  1 mon.magna102@0(leader).log v18374 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18374)
>2018-07-11 04:09:14.688649 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214437 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:14.838221 7f29149e4700  1 mon.magna102@0(leader).log v18375 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18375)
>2018-07-11 04:09:20.691212 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214440 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:20.839953 7f29149e4700  1 mon.magna102@0(leader).log v18376 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18376)
>2018-07-11 04:09:26.693920 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214443 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:26.840665 7f29149e4700  1 mon.magna102@0(leader).log v18377 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18377)
>2018-07-11 04:09:32.697019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214446 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:32.869648 7f29149e4700  1 mon.magna102@0(leader).log v18378 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18378)
>2018-07-11 04:09:37.736063 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214449 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:37.845523 7f29149e4700  1 mon.magna102@0(leader).log v18379 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18379)
>2018-07-11 04:09:42.737528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214452 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:42.854525 7f29149e4700  1 mon.magna102@0(leader).log v18380 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18380)
>2018-07-11 04:09:52.706235 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214455 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:52.856564 7f29149e4700  1 mon.magna102@0(leader).log v18381 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18381)
>2018-07-11 04:09:55.832013 7f29149e4700  1 mon.magna102@0(leader).log v18382 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18382)
>2018-07-11 04:09:57.739824 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214458 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:09:57.965721 7f29149e4700  1 mon.magna102@0(leader).log v18383 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18383)
>2018-07-11 04:10:00.099321 7f29149e4700  1 mon.magna102@0(leader).log v18384 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18384)
>2018-07-11 04:10:00.236198 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:10:03.850465 7f29149e4700  1 mon.magna102@0(leader).log v18385 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18385)
>2018-07-11 04:10:04.850511 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214461 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:05.000917 7f29149e4700  1 mon.magna102@0(leader).log v18386 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18386)
>2018-07-11 04:10:12.751253 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214464 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:12.868301 7f29149e4700  1 mon.magna102@0(leader).log v18387 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18387)
>2018-07-11 04:10:13.935085 7f29149e4700  1 mon.magna102@0(leader).log v18388 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18388)
>2018-07-11 04:10:17.751470 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214467 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:17.852303 7f29149e4700  1 mon.magna102@0(leader).log v18389 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18389)
>2018-07-11 04:10:22.760813 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214470 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:22.886307 7f29149e4700  1 mon.magna102@0(leader).log v18390 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18390)
>2018-07-11 04:10:23.953106 7f29149e4700  1 mon.magna102@0(leader).log v18391 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18391)
>2018-07-11 04:10:27.760983 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214473 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:27.886960 7f29149e4700  1 mon.magna102@0(leader).log v18392 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18392)
>2018-07-11 04:10:38.726293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214476 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:38.872807 7f29149e4700  1 mon.magna102@0(leader).log v18393 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18393)
>2018-07-11 04:10:47.764748 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214479 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:47.865170 7f29149e4700  1 mon.magna102@0(leader).log v18394 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18394)
>2018-07-11 04:10:52.765366 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214482 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:52.882485 7f29149e4700  1 mon.magna102@0(leader).log v18395 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18395)
>2018-07-11 04:10:57.765528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214485 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:10:57.866307 7f29149e4700  1 mon.magna102@0(leader).log v18396 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18396)
>2018-07-11 04:11:00.236382 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:11:06.738536 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214491 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:06.892189 7f29149e4700  1 mon.magna102@0(leader).log v18397 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18397)
>2018-07-11 04:11:16.743024 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214494 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:16.893130 7f29149e4700  1 mon.magna102@0(leader).log v18398 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18398)
>2018-07-11 04:11:22.745518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214497 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:22.893733 7f29149e4700  1 mon.magna102@0(leader).log v18399 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18399)
>2018-07-11 04:11:27.776767 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:27.885852 7f29149e4700  1 mon.magna102@0(leader).log v18400 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18400)
>2018-07-11 04:11:34.750725 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214503 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:34.904519 7f29149e4700  1 mon.magna102@0(leader).log v18401 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18401)
>2018-07-11 04:11:42.788239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214506 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:42.905287 7f29149e4700  1 mon.magna102@0(leader).log v18402 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18402)
>2018-07-11 04:11:47.864323 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10136. Immutable memtables: 0.
>
>2018-07-11 04:11:47.864367 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 673] Syncing log #10133
>2018-07-11 04:11:47.880785 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.864357) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:11:47.880796 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 673] Flushing memtable with next log file: 10136
>
>2018-07-11 04:11:47.880822 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282307880814, "job": 673, "event": "flush_started", "num_memtables": 1, "num_entries": 1409, "num_deletes": 250, "memory_usage": 2532856}
>2018-07-11 04:11:47.880825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 673] Level-0 flush table #10137: started
>2018-07-11 04:11:47.939675 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282307939660, "cf_name": "default", "job": 673, "event": "table_file_creation", "file_number": 10137, "file_size": 1678568, "table_properties": {"data_size": 1664985, "index_size": 6806, "filter_size": 5857, "raw_key_size": 10722, "raw_average_key_size": 21, "raw_value_size": 1653088, "raw_average_value_size": 3266, "num_data_blocks": 210, "num_entries": 506, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:11:47.939685 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 673] Level-0 flush table #10137: 1678568 bytes OK
>2018-07-11 04:11:47.989682 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.956207) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10137 started
>2018-07-11 04:11:47.989690 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.989646) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10137: memtable #1 done
>2018-07-11 04:11:47.989693 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.989660) EVENT_LOG_v1 {"time_micros": 1531282307989654, "job": 673, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:11:47.989695 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.989673) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:11:47.989715 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 673] Try to delete WAL files size 2504826, prev total WAL file size 2504826, number of live WAL files 2.
>
>2018-07-11 04:11:47.990396 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:11:47.990448 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:11:47.990441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:11:47.990457 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 674] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:11:47.990463 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 673 Base level 0, inputs: [10137(1639KB)], [10135(13MB)]
>
>2018-07-11 04:11:47.990476 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282307990466, "job": 674, "event": "compaction_started", "files_L0": [10137], "files_L1": [10135], "score": -1, "input_data_size": 16036154}
>2018-07-11 04:11:48.173733 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 674] Generated table #10138: 3219 keys, 15088717 bytes
>2018-07-11 04:11:48.173782 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282308173754, "cf_name": "default", "job": 674, "event": "table_file_creation", "file_number": 10138, "file_size": 15088717, "table_properties": {"data_size": 14972820, "index_size": 66060, "filter_size": 48910, "raw_key_size": 64738, "raw_average_key_size": 20, "raw_value_size": 14880632, "raw_average_value_size": 4622, "num_data_blocks": 1935, "num_entries": 3219, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:11:48.215246 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 674] Compacted 1@0 + 1@1 files to L1 => 15088717 bytes
>2018-07-11 04:11:48.248769 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:11:48.248703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.3 rd, 67.1 wr, level 1, files in(1, 1) out(1) MB in(1.6, 13.7) out(14.4), read-write-amplify(18.5) write-amplify(9.0) OK, records in: 3615, records dropped: 396
>
>2018-07-11 04:11:48.248778 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:11:48.248724) EVENT_LOG_v1 {"time_micros": 1531282308248715, "job": 674, "event": "compaction_finished", "compaction_time_micros": 224755, "output_level": 1, "num_output_files": 1, "total_output_size": 15088717, "num_input_records": 3615, "num_output_records": 3219, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:11:48.249215 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282308249211, "job": 674, "event": "table_file_deletion", "file_number": 10137}
>2018-07-11 04:11:48.251862 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282308251859, "job": 674, "event": "table_file_deletion", "file_number": 10135}
>2018-07-11 04:11:48.251977 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:11:48.864215 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214509 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:49.016273 7f29149e4700  1 mon.magna102@0(leader).log v18403 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18403)
>2018-07-11 04:11:54.759600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214512 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:11:54.908649 7f29149e4700  1 mon.magna102@0(leader).log v18404 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18404)
>2018-07-11 04:12:00.236551 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:12:02.831172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214515 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:03.892810 7f29149e4700  1 mon.magna102@0(leader).log v18405 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18405)
>2018-07-11 04:12:07.831415 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214518 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:07.901643 7f29149e4700  1 mon.magna102@0(leader).log v18406 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18406)
>2018-07-11 04:12:17.831813 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214521 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:17.903762 7f29149e4700  1 mon.magna102@0(leader).log v18407 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18407)
>2018-07-11 04:12:22.832052 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214524 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:22.954449 7f29149e4700  1 mon.magna102@0(leader).log v18408 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18408)
>2018-07-11 04:12:27.832276 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214527 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:27.904987 7f29149e4700  1 mon.magna102@0(leader).log v18409 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18409)
>2018-07-11 04:12:32.832513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214530 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:32.955578 7f29149e4700  1 mon.magna102@0(leader).log v18410 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18410)
>2018-07-11 04:12:38.778896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214533 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:38.931230 7f29149e4700  1 mon.magna102@0(leader).log v18411 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18411)
>2018-07-11 04:12:47.833079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214536 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:47.915270 7f29149e4700  1 mon.magna102@0(leader).log v18412 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18412)
>2018-07-11 04:12:52.833334 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214539 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:52.957672 7f29149e4700  1 mon.magna102@0(leader).log v18413 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18413)
>2018-07-11 04:12:57.833530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214542 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:12:57.933216 7f29149e4700  1 mon.magna102@0(leader).log v18414 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18414)
>2018-07-11 04:13:00.236741 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:13:07.833879 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214545 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:07.959089 7f29149e4700  1 mon.magna102@0(leader).log v18415 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18415)
>2018-07-11 04:13:12.834384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214548 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:12.952670 7f29149e4700  1 mon.magna102@0(leader).log v18416 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18416)
>2018-07-11 04:13:17.834578 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214551 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:17.928119 7f29149e4700  1 mon.magna102@0(leader).log v18417 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18417)
>2018-07-11 04:13:27.834970 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214554 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:27.929019 7f29149e4700  1 mon.magna102@0(leader).log v18418 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18418)
>2018-07-11 04:13:32.837667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214557 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:32.954699 7f29149e4700  1 mon.magna102@0(leader).log v18419 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18419)
>2018-07-11 04:13:37.837819 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214563 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:37.938471 7f29149e4700  1 mon.magna102@0(leader).log v18420 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18420)
>2018-07-11 04:13:44.807582 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214566 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:44.955978 7f29149e4700  1 mon.magna102@0(leader).log v18421 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18421)
>2018-07-11 04:13:52.811236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214569 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:52.965109 7f29149e4700  1 mon.magna102@0(leader).log v18422 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18422)
>2018-07-11 04:13:58.813771 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214572 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:13:58.965677 7f29149e4700  1 mon.magna102@0(leader).log v18423 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18423)
>2018-07-11 04:14:00.236924 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:14:04.949796 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10139. Immutable memtables: 0.
>
>2018-07-11 04:14:04.949840 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 675] Syncing log #10136
>2018-07-11 04:14:04.966338 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:14:04.949831) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:14:04.966349 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 675] Flushing memtable with next log file: 10139
>
>2018-07-11 04:14:04.966376 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282444966368, "job": 675, "event": "flush_started", "num_memtables": 1, "num_entries": 907, "num_deletes": 251, "memory_usage": 1425240}
>2018-07-11 04:14:04.966379 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 675] Level-0 flush table #10140: started
>2018-07-11 04:14:05.016891 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445016876, "cf_name": "default", "job": 675, "event": "table_file_creation", "file_number": 10140, "file_size": 1370980, "table_properties": {"data_size": 1359068, "index_size": 6015, "filter_size": 4977, "raw_key_size": 9426, "raw_average_key_size": 20, "raw_value_size": 1348379, "raw_average_value_size": 2924, "num_data_blocks": 183, "num_entries": 461, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:14:05.016902 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 675] Level-0 flush table #10140: 1370980 bytes OK
>2018-07-11 04:14:05.068120 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.034702) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10140 started
>2018-07-11 04:14:05.068128 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.068084) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10140: memtable #1 done
>2018-07-11 04:14:05.068131 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.068099) EVENT_LOG_v1 {"time_micros": 1531282445068092, "job": 675, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:14:05.068133 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.068110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:14:05.068152 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 675] Try to delete WAL files size 1405912, prev total WAL file size 1405912, number of live WAL files 2.
>
>2018-07-11 04:14:05.068574 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:14:05.068625 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.068617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:14:05.068634 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 676] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:14:05.068640 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 675 Base level 0, inputs: [10140(1338KB)], [10138(14MB)]
>
>2018-07-11 04:14:05.068654 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445068644, "job": 676, "event": "compaction_started", "files_L0": [10140], "files_L1": [10138], "score": -1, "input_data_size": 16459697}
>2018-07-11 04:14:05.243860 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 676] Generated table #10141: 3170 keys, 14456483 bytes
>2018-07-11 04:14:05.243905 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445243878, "cf_name": "default", "job": 676, "event": "table_file_creation", "file_number": 10141, "file_size": 14456483, "table_properties": {"data_size": 14344464, "index_size": 63987, "filter_size": 47105, "raw_key_size": 63917, "raw_average_key_size": 20, "raw_value_size": 14254360, "raw_average_value_size": 4496, "num_data_blocks": 1866, "num_entries": 3170, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:14:05.276996 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 676] Compacted 1@0 + 1@1 files to L1 => 14456483 bytes
>2018-07-11 04:14:05.310500 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.310432) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.0 rd, 69.4 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.4) out(13.8), read-write-amplify(22.6) write-amplify(10.5) OK, records in: 3680, records dropped: 510
>
>2018-07-11 04:14:05.310509 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:14:05.310455) EVENT_LOG_v1 {"time_micros": 1531282445310445, "job": 676, "event": "compaction_finished", "compaction_time_micros": 208326, "output_level": 1, "num_output_files": 1, "total_output_size": 14456483, "num_input_records": 3680, "num_output_records": 3170, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:14:05.310859 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445310855, "job": 676, "event": "table_file_deletion", "file_number": 10140}
>2018-07-11 04:14:05.313864 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445313861, "job": 676, "event": "table_file_deletion", "file_number": 10138}
>2018-07-11 04:14:07.849258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214575 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:07.944662 7f29149e4700  1 mon.magna102@0(leader).log v18424 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18424)
>2018-07-11 04:14:16.821827 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214581 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:16.987408 7f29149e4700  1 mon.magna102@0(leader).log v18425 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18425)
>2018-07-11 04:14:26.826195 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214584 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:26.988204 7f29149e4700  1 mon.magna102@0(leader).log v18426 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18426)
>2018-07-11 04:14:32.828756 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214587 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:32.980578 7f29149e4700  1 mon.magna102@0(leader).log v18427 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18427)
>2018-07-11 04:14:38.831227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214590 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:38.989480 7f29149e4700  1 mon.magna102@0(leader).log v18428 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18428)
>2018-07-11 04:14:47.864719 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214593 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:47.998545 7f29149e4700  1 mon.magna102@0(leader).log v18429 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18429)
>2018-07-11 04:14:57.873988 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214596 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:14:57.984002 7f29149e4700  1 mon.magna102@0(leader).log v18430 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18430)
>2018-07-11 04:15:00.237115 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:15:02.875965 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214599 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:02.992990 7f29149e4700  1 mon.magna102@0(leader).log v18431 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18431)
>2018-07-11 04:15:07.876115 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214605 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:07.976773 7f29149e4700  1 mon.magna102@0(leader).log v18432 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18432)
>2018-07-11 04:15:17.877265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214608 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:17.986092 7f29149e4700  1 mon.magna102@0(leader).log v18433 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18433)
>2018-07-11 04:15:26.851730 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214614 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:27.012000 7f29149e4700  1 mon.magna102@0(leader).log v18434 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18434)
>2018-07-11 04:15:34.855079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214617 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:35.004475 7f29149e4700  1 mon.magna102@0(leader).log v18435 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18435)
>2018-07-11 04:15:42.896550 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214620 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:43.013556 7f29149e4700  1 mon.magna102@0(leader).log v18436 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18436)
>2018-07-11 04:15:47.896738 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214623 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:47.997367 7f29149e4700  1 mon.magna102@0(leader).log v18437 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18437)
>2018-07-11 04:15:57.897076 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214626 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:15:58.007820 7f29149e4700  1 mon.magna102@0(leader).log v18438 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18438)
>2018-07-11 04:16:00.237312 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:16:02.908157 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214629 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:03.041832 7f29149e4700  1 mon.magna102@0(leader).log v18439 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18439)
>2018-07-11 04:16:07.908373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214632 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:08.025597 7f29149e4700  1 mon.magna102@0(leader).log v18440 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18440)
>2018-07-11 04:16:16.873079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214638 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:17.051509 7f29149e4700  1 mon.magna102@0(leader).log v18441 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18441)
>2018-07-11 04:16:26.877268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214641 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:27.044009 7f29149e4700  1 mon.magna102@0(leader).log v18442 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18442)
>2018-07-11 04:16:32.919293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214644 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:33.053003 7f29149e4700  1 mon.magna102@0(leader).log v18443 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18443)
>2018-07-11 04:16:37.919513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214647 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:38.045112 7f29149e4700  1 mon.magna102@0(leader).log v18444 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18444)
>2018-07-11 04:16:47.919852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214650 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:48.021002 7f29149e4700  1 mon.magna102@0(leader).log v18445 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18445)
>2018-07-11 04:16:52.921303 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214653 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:53.039452 7f29149e4700  1 mon.magna102@0(leader).log v18446 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18446)
>2018-07-11 04:16:57.921493 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214656 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:16:58.023362 7f29149e4700  1 mon.magna102@0(leader).log v18447 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18447)
>2018-07-11 04:17:00.237485 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:17:06.894146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214662 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:07.049252 7f29149e4700  1 mon.magna102@0(leader).log v18448 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18448)
>2018-07-11 04:17:12.932841 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214665 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:13.049882 7f29149e4700  1 mon.magna102@0(leader).log v18449 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18449)
>2018-07-11 04:17:20.900007 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214668 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:21.050560 7f29149e4700  1 mon.magna102@0(leader).log v18450 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18450)
>2018-07-11 04:17:27.959171 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214671 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:28.051375 7f29149e4700  1 mon.magna102@0(leader).log v18451 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18451)
>2018-07-11 04:17:32.959411 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214674 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:33.085387 7f29149e4700  1 mon.magna102@0(leader).log v18452 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18452)
>2018-07-11 04:17:38.907724 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214677 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:39.061028 7f29149e4700  1 mon.magna102@0(leader).log v18453 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18453)
>2018-07-11 04:17:47.959953 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214680 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:48.070120 7f29149e4700  1 mon.magna102@0(leader).log v18454 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18454)
>2018-07-11 04:17:57.960325 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214683 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:17:58.047265 7f29149e4700  1 mon.magna102@0(leader).log v18455 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18455)
>2018-07-11 04:18:00.237660 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:18:02.960554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214686 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:03.072892 7f29149e4700  1 mon.magna102@0(leader).log v18456 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18456)
>2018-07-11 04:18:07.960709 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214689 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:08.056687 7f29149e4700  1 mon.magna102@0(leader).log v18457 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18457)
>2018-07-11 04:18:09.131821 7f29149e4700  1 mon.magna102@0(leader).log v18458 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18458)
>2018-07-11 04:18:14.922835 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214695 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:15.074351 7f29149e4700  1 mon.magna102@0(leader).log v18459 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18459)
>2018-07-11 04:18:18.782886 7f29149e4700  1 mon.magna102@0(leader).log v18460 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18460)
>2018-07-11 04:18:22.966912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214698 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:23.083940 7f29149e4700  1 mon.magna102@0(leader).log v18461 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18461)
>2018-07-11 04:18:28.928874 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214701 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:29.076208 7f29149e4700  1 mon.magna102@0(leader).log v18462 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18462)
>2018-07-11 04:18:37.968138 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214704 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:38.068683 7f29149e4700  1 mon.magna102@0(leader).log v18463 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18463)
>2018-07-11 04:18:47.969172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214707 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:48.062471 7f29149e4700  1 mon.magna102@0(leader).log v18464 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18464)
>2018-07-11 04:18:52.971244 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214713 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:53.088301 7f29149e4700  1 mon.magna102@0(leader).log v18465 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18465)
>2018-07-11 04:18:57.971467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214716 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:18:58.088721 7f29149e4700  1 mon.magna102@0(leader).log v18466 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18466)
>2018-07-11 04:19:00.237843 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:19:06.944776 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214719 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:07.097953 7f29149e4700  1 mon.magna102@0(leader).log v18467 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18467)
>2018-07-11 04:19:12.989874 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214722 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:13.123595 7f29149e4700  1 mon.magna102@0(leader).log v18468 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18468)
>2018-07-11 04:19:17.990057 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214725 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:18.115702 7f29149e4700  1 mon.magna102@0(leader).log v18469 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18469)
>2018-07-11 04:19:24.952378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214728 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:25.124879 7f29149e4700  1 mon.magna102@0(leader).log v18470 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18470)
>2018-07-11 04:19:30.955038 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214731 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:31.125487 7f29149e4700  1 mon.magna102@0(leader).log v18471 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18471)
>2018-07-11 04:19:36.957566 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214734 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:37.126061 7f29149e4700  1 mon.magna102@0(leader).log v18472 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18472)
>2018-07-11 04:19:43.000846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214737 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:43.119619 7f29149e4700  1 mon.magna102@0(leader).log v18473 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18473)
>2018-07-11 04:19:48.001074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214740 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:48.153665 7f29149e4700  1 mon.magna102@0(leader).log v18474 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18474)
>2018-07-11 04:19:53.036822 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214743 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:19:54.095968 7f29149e4700  1 mon.magna102@0(leader).log v18475 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18475)
>2018-07-11 04:20:00.238031 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:20:00.967693 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214746 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:01.121725 7f29149e4700  1 mon.magna102@0(leader).log v18476 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18476)
>2018-07-11 04:20:08.037435 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214749 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:08.106257 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10142. Immutable memtables: 0.
>
>2018-07-11 04:20:08.106305 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 677] Syncing log #10139
>2018-07-11 04:20:08.122572 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.106291) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:20:08.122583 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 677] Flushing memtable with next log file: 10142
>
>2018-07-11 04:20:08.122610 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808122601, "job": 677, "event": "flush_started", "num_memtables": 1, "num_entries": 1953, "num_deletes": 250, "memory_usage": 3715704}
>2018-07-11 04:20:08.122613 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 677] Level-0 flush table #10143: started
>2018-07-11 04:20:08.214913 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808214898, "cf_name": "default", "job": 677, "event": "table_file_creation", "file_number": 10143, "file_size": 2388517, "table_properties": {"data_size": 2369851, "index_size": 9679, "filter_size": 8063, "raw_key_size": 12804, "raw_average_key_size": 21, "raw_value_size": 2354429, "raw_average_value_size": 3866, "num_data_blocks": 298, "num_entries": 609, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:20:08.214923 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 677] Level-0 flush table #10143: 2388517 bytes OK
>2018-07-11 04:20:08.249097 7f29149e4700  1 mon.magna102@0(leader).log v18477 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18477)
>2018-07-11 04:20:08.281555 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.248133) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10143 started
>2018-07-11 04:20:08.281563 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.281507) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10143: memtable #1 done
>2018-07-11 04:20:08.281566 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.281522) EVENT_LOG_v1 {"time_micros": 1531282808281515, "job": 677, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:20:08.281568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.281545) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:20:08.281574 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 677] Try to delete WAL files size 3677854, prev total WAL file size 3696262, number of live WAL files 2.
>
>2018-07-11 04:20:08.282517 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:20:08.282709 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 04:20:08.282716 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 44835.1 total, 737.4 interval
>Cumulative writes: 63K writes, 302K keys, 63K commit groups, 1.0 writes per commit group, ingest: 0.51 GB, 0.01 MB/s
>Cumulative WAL: 63K writes, 63K syncs, 1.00 writes per sync, written: 0.51 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 976 writes, 4277 keys, 976 commit groups, 1.0 writes per commit group, ingest: 7.25 MB, 0.01 MB/s
>Interval WAL: 976 writes, 976 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.28 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        28       339    0.083       0      0
>  L1      1/1   13.79 MB   0.0      5.3     0.4      4.9       4.9     -0.0       0.0  10.9     72.4     66.2        75       337    0.223   1191K   140K
> Sum      2/2   16.06 MB   0.0      5.3     0.4      4.9       5.3      0.4       0.0  11.8     52.6     52.6       103       676    0.153   1191K   140K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.1     51.6     52.2         1         6    0.150     10K   1416
>Uptime(secs): 44835.1 total, 44835.1 interval
>Flush(GB): cumulative 0.449, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.31 GB write, 0.12 MB/s write, 5.31 GB read, 0.12 MB/s read, 103.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.28 MB   0.0      0.0     0.0      0.0       0.4      0.4       0.0   1.0      0.0     16.3        28       339    0.083       0      0
>  L1      1/1   13.79 MB   0.0      5.3     0.4      4.9       4.9     -0.0       0.0  10.9     72.4     66.2        75       337    0.223   1191K   140K
> Sum      2/2   16.06 MB   0.0      5.3     0.4      4.9       5.3      0.4       0.0  11.8     52.6     52.6       103       676    0.153   1191K   140K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 44835.1 total, 44835.1 interval
>Flush(GB): cumulative 0.449, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.31 GB write, 0.12 MB/s write, 5.31 GB read, 0.12 MB/s read, 103.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 04:20:08.282737 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.282731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:20:08.282743 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 678] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:20:08.282749 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 677 Base level 0, inputs: [10143(2332KB)], [10141(13MB)]
>
>2018-07-11 04:20:08.282761 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808282752, "job": 678, "event": "compaction_started", "files_L0": [10143], "files_L1": [10141], "score": -1, "input_data_size": 16845000}
>2018-07-11 04:20:08.474005 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 678] Generated table #10144: 3444 keys, 16327688 bytes
>2018-07-11 04:20:08.474052 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808474025, "cf_name": "default", "job": 678, "event": "table_file_creation", "file_number": 10144, "file_size": 16327688, "table_properties": {"data_size": 16202609, "index_size": 71305, "filter_size": 52847, "raw_key_size": 69308, "raw_average_key_size": 20, "raw_value_size": 16103560, "raw_average_value_size": 4675, "num_data_blocks": 2088, "num_entries": 3444, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:20:08.507199 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 678] Compacted 1@0 + 1@1 files to L1 => 16327688 bytes
>2018-07-11 04:20:08.540695 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.540628) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.1 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(2.3, 13.8) out(15.6), read-write-amplify(13.9) write-amplify(6.8) OK, records in: 3779, records dropped: 335
>
>2018-07-11 04:20:08.540705 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.540649) EVENT_LOG_v1 {"time_micros": 1531282808540640, "job": 678, "event": "compaction_finished", "compaction_time_micros": 224421, "output_level": 1, "num_output_files": 1, "total_output_size": 16327688, "num_input_records": 3779, "num_output_records": 3444, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:20:08.541289 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808541285, "job": 678, "event": "table_file_deletion", "file_number": 10143}
>2018-07-11 04:20:08.544040 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808544037, "job": 678, "event": "table_file_deletion", "file_number": 10141}
>2018-07-11 04:20:08.544372 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10145. Immutable memtables: 0.
>
>2018-07-11 04:20:08.544411 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 679] Syncing log #10142
>2018-07-11 04:20:08.557721 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.544403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:20:08.557732 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 679] Flushing memtable with next log file: 10145
>
>2018-07-11 04:20:08.557758 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808557751, "job": 679, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20736}
>2018-07-11 04:20:08.557761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 679] Level-0 flush table #10146: started
>2018-07-11 04:20:08.599873 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808599860, "cf_name": "default", "job": 679, "event": "table_file_creation", "file_number": 10146, "file_size": 19523, "table_properties": {"data_size": 18457, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18234, "raw_average_value_size": 2279, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:20:08.599883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 679] Level-0 flush table #10146: 19523 bytes OK
>2018-07-11 04:20:08.649919 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.616405) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10146 started
>2018-07-11 04:20:08.649927 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.649885) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10146: memtable #1 done
>2018-07-11 04:20:08.649929 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.649899) EVENT_LOG_v1 {"time_micros": 1531282808649893, "job": 679, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:20:08.649945 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.649910) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:20:08.649950 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 679] Try to delete WAL files size 18408, prev total WAL file size 18408, number of live WAL files 2.
>
>2018-07-11 04:20:08.650047 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:20:08.650092 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.650086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:20:08.650101 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 680] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:20:08.650107 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 679 Base level 0, inputs: [10146(19KB)], [10144(15MB)]
>
>2018-07-11 04:20:08.650120 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808650110, "job": 680, "event": "compaction_started", "files_L0": [10146], "files_L1": [10144], "score": -1, "input_data_size": 16347211}
>2018-07-11 04:20:08.842334 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 680] Generated table #10147: 3447 keys, 16346098 bytes
>2018-07-11 04:20:08.842381 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808842364, "cf_name": "default", "job": 680, "event": "table_file_creation", "file_number": 10147, "file_size": 16346098, "table_properties": {"data_size": 16220893, "index_size": 71374, "filter_size": 52904, "raw_key_size": 69369, "raw_average_key_size": 20, "raw_value_size": 16121754, "raw_average_value_size": 4677, "num_data_blocks": 2090, "num_entries": 3447, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:20:08.875516 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 680] Compacted 1@0 + 1@1 files to L1 => 16346098 bytes
>2018-07-11 04:20:08.909029 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.908962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.5 rd, 72.5 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.6) out(15.6), read-write-amplify(1674.6) write-amplify(837.3) OK, records in: 3452, records dropped: 5
>
>2018-07-11 04:20:08.909038 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:08.908984) EVENT_LOG_v1 {"time_micros": 1531282808908975, "job": 680, "event": "compaction_finished", "compaction_time_micros": 225380, "output_level": 1, "num_output_files": 1, "total_output_size": 16346098, "num_input_records": 3452, "num_output_records": 3447, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:20:08.909133 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808909128, "job": 680, "event": "table_file_deletion", "file_number": 10146}
>2018-07-11 04:20:08.912257 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282808912254, "job": 680, "event": "table_file_deletion", "file_number": 10144}
>2018-07-11 04:20:13.072641 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214752 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:14.111419 7f29149e4700  1 mon.magna102@0(leader).log v18478 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18478)
>2018-07-11 04:20:18.975922 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214755 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:19.128701 7f29149e4700  1 mon.magna102@0(leader).log v18479 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18479)
>2018-07-11 04:20:26.979379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214758 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:27.129570 7f29149e4700  1 mon.magna102@0(leader).log v18480 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18480)
>2018-07-11 04:20:27.196649 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10148. Immutable memtables: 0.
>
>2018-07-11 04:20:27.196716 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 681] Syncing log #10145
>2018-07-11 04:20:27.213329 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.196706) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:20:27.213342 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 681] Flushing memtable with next log file: 10148
>
>2018-07-11 04:20:27.213355 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827213348, "job": 681, "event": "flush_started", "num_memtables": 1, "num_entries": 350, "num_deletes": 251, "memory_usage": 225536}
>2018-07-11 04:20:27.213358 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 681] Level-0 flush table #10149: started
>2018-07-11 04:20:27.255472 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827255460, "cf_name": "default", "job": 681, "event": "table_file_creation", "file_number": 10149, "file_size": 212987, "table_properties": {"data_size": 209864, "index_size": 903, "filter_size": 1301, "raw_key_size": 5850, "raw_average_key_size": 20, "raw_value_size": 205310, "raw_average_value_size": 710, "num_data_blocks": 28, "num_entries": 289, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:20:27.255482 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 681] Level-0 flush table #10149: 212987 bytes OK
>2018-07-11 04:20:27.305510 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.271999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10149 started
>2018-07-11 04:20:27.305518 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.305476) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10149: memtable #1 done
>2018-07-11 04:20:27.305521 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.305491) EVENT_LOG_v1 {"time_micros": 1531282827305484, "job": 681, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:20:27.305535 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.305501) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:20:27.305541 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 681] Try to delete WAL files size 216275, prev total WAL file size 216275, number of live WAL files 2.
>
>2018-07-11 04:20:27.305677 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:20:27.305726 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.305719) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:20:27.305735 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 682] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:20:27.305741 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 681 Base level 0, inputs: [10149(207KB)], [10147(15MB)]
>
>2018-07-11 04:20:27.305754 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827305744, "job": 682, "event": "compaction_started", "files_L0": [10149], "files_L1": [10147], "score": -1, "input_data_size": 16559085}
>2018-07-11 04:20:27.489539 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 682] Generated table #10150: 3226 keys, 14549352 bytes
>2018-07-11 04:20:27.489582 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827489556, "cf_name": "default", "job": 682, "event": "table_file_creation", "file_number": 10150, "file_size": 14549352, "table_properties": {"data_size": 14436977, "index_size": 64037, "filter_size": 47411, "raw_key_size": 64972, "raw_average_key_size": 20, "raw_value_size": 14346057, "raw_average_value_size": 4447, "num_data_blocks": 1866, "num_entries": 3226, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:20:27.531063 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 682] Compacted 1@0 + 1@1 files to L1 => 14549352 bytes
>2018-07-11 04:20:27.564605 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.564533) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.5 rd, 64.6 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.6) out(13.9), read-write-amplify(146.1) write-amplify(68.3) OK, records in: 3736, records dropped: 510
>
>2018-07-11 04:20:27.564614 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:20:27.564555) EVENT_LOG_v1 {"time_micros": 1531282827564545, "job": 682, "event": "compaction_finished", "compaction_time_micros": 225294, "output_level": 1, "num_output_files": 1, "total_output_size": 14549352, "num_input_records": 3736, "num_output_records": 3226, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:20:27.564742 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827564738, "job": 682, "event": "table_file_deletion", "file_number": 10149}
>2018-07-11 04:20:27.567849 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282827567847, "job": 682, "event": "table_file_deletion", "file_number": 10147}
>2018-07-11 04:20:33.073402 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214761 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:34.132258 7f29149e4700  1 mon.magna102@0(leader).log v18481 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18481)
>2018-07-11 04:20:38.984564 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214764 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:39.133021 7f29149e4700  1 mon.magna102@0(leader).log v18482 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18482)
>2018-07-11 04:20:48.073963 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214767 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:48.192075 7f29149e4700  1 mon.magna102@0(leader).log v18483 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18483)
>2018-07-11 04:20:53.074205 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214770 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:54.127258 7f29149e4700  1 mon.magna102@0(leader).log v18484 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18484)
>2018-07-11 04:20:58.074373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214773 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:20:58.194412 7f29149e4700  1 mon.magna102@0(leader).log v18485 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18485)
>2018-07-11 04:21:00.238214 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:21:03.074597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214776 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:04.136813 7f29149e4700  1 mon.magna102@0(leader).log v18486 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18486)
>2018-07-11 04:21:08.074821 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214779 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:08.195617 7f29149e4700  1 mon.magna102@0(leader).log v18487 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18487)
>2018-07-11 04:21:14.999769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214782 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:15.154668 7f29149e4700  1 mon.magna102@0(leader).log v18488 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18488)
>2018-07-11 04:21:23.075397 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214785 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:24.138782 7f29149e4700  1 mon.magna102@0(leader).log v18489 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18489)
>2018-07-11 04:21:28.075598 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214788 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:28.197570 7f29149e4700  1 mon.magna102@0(leader).log v18490 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18490)
>2018-07-11 04:21:33.075827 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214791 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:34.139910 7f29149e4700  1 mon.magna102@0(leader).log v18491 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18491)
>2018-07-11 04:21:38.076064 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214794 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:38.148751 7f29149e4700  1 mon.magna102@0(leader).log v18492 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18492)
>2018-07-11 04:21:43.076292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214797 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:44.141122 7f29149e4700  1 mon.magna102@0(leader).log v18493 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18493)
>2018-07-11 04:21:53.016239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214800 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:53.193254 7f29149e4700  1 mon.magna102@0(leader).log v18494 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18494)
>2018-07-11 04:21:58.076820 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214803 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:21:58.177011 7f29149e4700  1 mon.magna102@0(leader).log v18495 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18495)
>2018-07-11 04:22:00.238398 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:22:08.077215 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214806 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:08.152918 7f29149e4700  1 mon.magna102@0(leader).log v18496 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18496)
>2018-07-11 04:22:13.077462 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214809 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:13.195372 7f29149e4700  1 mon.magna102@0(leader).log v18497 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18497)
>2018-07-11 04:22:18.077666 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214812 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:18.154153 7f29149e4700  1 mon.magna102@0(leader).log v18498 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18498)
>2018-07-11 04:22:23.077908 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214815 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:23.196462 7f29149e4700  1 mon.magna102@0(leader).log v18499 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18499)
>2018-07-11 04:22:29.031205 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214818 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:29.180387 7f29149e4700  1 mon.magna102@0(leader).log v18500 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18500)
>2018-07-11 04:22:38.078441 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214821 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:38.172780 7f29149e4700  1 mon.magna102@0(leader).log v18501 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18501)
>2018-07-11 04:22:43.078678 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214824 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:43.199657 7f29149e4700  1 mon.magna102@0(leader).log v18502 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18502)
>2018-07-11 04:22:48.078911 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214827 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:48.183538 7f29149e4700  1 mon.magna102@0(leader).log v18503 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18503)
>2018-07-11 04:22:57.043017 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214833 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:22:57.192682 7f29149e4700  1 mon.magna102@0(leader).log v18504 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18504)
>2018-07-11 04:23:00.238583 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:23:07.047212 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214836 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:07.201990 7f29149e4700  1 mon.magna102@0(leader).log v18505 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18505)
>2018-07-11 04:23:13.085576 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214839 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:13.202618 7f29149e4700  1 mon.magna102@0(leader).log v18506 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18506)
>2018-07-11 04:23:18.085769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214842 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:18.186393 7f29149e4700  1 mon.magna102@0(leader).log v18507 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18507)
>2018-07-11 04:23:28.095225 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214845 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:28.212383 7f29149e4700  1 mon.magna102@0(leader).log v18508 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18508)
>2018-07-11 04:23:33.096016 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214848 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:33.229711 7f29149e4700  1 mon.magna102@0(leader).log v18509 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18509)
>2018-07-11 04:23:38.096204 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214851 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:38.230154 7f29149e4700  1 mon.magna102@0(leader).log v18510 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18510)
>2018-07-11 04:23:48.098521 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214854 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:48.198955 7f29149e4700  1 mon.magna102@0(leader).log v18511 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18511)
>2018-07-11 04:23:53.099264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214857 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:53.216326 7f29149e4700  1 mon.magna102@0(leader).log v18512 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18512)
>2018-07-11 04:23:58.099455 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214860 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:23:58.200075 7f29149e4700  1 mon.magna102@0(leader).log v18513 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18513)
>2018-07-11 04:24:00.238758 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:24:05.071912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214863 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:05.225944 7f29149e4700  1 mon.magna102@0(leader).log v18514 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18514)
>2018-07-11 04:24:13.118045 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214866 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:13.235042 7f29149e4700  1 mon.magna102@0(leader).log v18515 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18515)
>2018-07-11 04:24:18.118208 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214869 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:18.194342 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10151. Immutable memtables: 0.
>
>2018-07-11 04:24:18.194387 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 683] Syncing log #10148
>2018-07-11 04:24:18.210582 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.194377) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:24:18.210593 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 683] Flushing memtable with next log file: 10151
>
>2018-07-11 04:24:18.210607 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058210599, "job": 683, "event": "flush_started", "num_memtables": 1, "num_entries": 1590, "num_deletes": 500, "memory_usage": 2413432}
>2018-07-11 04:24:18.210609 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 683] Level-0 flush table #10152: started
>2018-07-11 04:24:18.302959 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058302944, "cf_name": "default", "job": 683, "event": "table_file_creation", "file_number": 10152, "file_size": 2320238, "table_properties": {"data_size": 2300688, "index_size": 10065, "filter_size": 8560, "raw_key_size": 17441, "raw_average_key_size": 20, "raw_value_size": 2281914, "raw_average_value_size": 2703, "num_data_blocks": 305, "num_entries": 844, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 04:24:18.302969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 683] Level-0 flush table #10152: 2320238 bytes OK
>2018-07-11 04:24:18.336370 7f29149e4700  1 mon.magna102@0(leader).log v18516 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18516)
>2018-07-11 04:24:18.369545 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.336133) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10152 started
>2018-07-11 04:24:18.369555 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.369507) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10152: memtable #1 done
>2018-07-11 04:24:18.369557 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.369522) EVENT_LOG_v1 {"time_micros": 1531283058369515, "job": 683, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:24:18.369560 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.369535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:24:18.369567 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 683] Try to delete WAL files size 2379483, prev total WAL file size 2398183, number of live WAL files 2.
>
>2018-07-11 04:24:18.370192 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:24:18.370253 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.370245) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 04:24:18.370265 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 684] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:24:18.370272 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 683 Base level 0, inputs: [10152(2265KB)], [10150(13MB)]
>
>2018-07-11 04:24:18.370286 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058370276, "job": 684, "event": "compaction_started", "files_L0": [10152], "files_L1": [10150], "score": -1, "input_data_size": 16869590}
>2018-07-11 04:24:18.553672 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 684] Generated table #10153: 3062 keys, 14586201 bytes
>2018-07-11 04:24:18.553719 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058553692, "cf_name": "default", "job": 684, "event": "table_file_creation", "file_number": 10153, "file_size": 14586201, "table_properties": {"data_size": 14472919, "index_size": 64964, "filter_size": 47391, "raw_key_size": 61707, "raw_average_key_size": 20, "raw_value_size": 14383716, "raw_average_value_size": 4697, "num_data_blocks": 1906, "num_entries": 3062, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:24:18.595196 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 684] Compacted 1@0 + 1@1 files to L1 => 14586201 bytes
>2018-07-11 04:24:18.628704 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.628633) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.0 rd, 64.9 wr, level 1, files in(1, 1) out(1) MB in(2.2, 13.9) out(13.9), read-write-amplify(13.6) write-amplify(6.3) OK, records in: 4070, records dropped: 1008
>
>2018-07-11 04:24:18.628713 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.628654) EVENT_LOG_v1 {"time_micros": 1531283058628645, "job": 684, "event": "compaction_finished", "compaction_time_micros": 224892, "output_level": 1, "num_output_files": 1, "total_output_size": 14586201, "num_input_records": 4070, "num_output_records": 3062, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:24:18.629248 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058629244, "job": 684, "event": "table_file_deletion", "file_number": 10152}
>2018-07-11 04:24:18.632015 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058632012, "job": 684, "event": "table_file_deletion", "file_number": 10150}
>2018-07-11 04:24:18.632279 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10154. Immutable memtables: 0.
>
>2018-07-11 04:24:18.632350 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 685] Syncing log #10151
>2018-07-11 04:24:18.645737 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.632340) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:24:18.645750 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 685] Flushing memtable with next log file: 10154
>
>2018-07-11 04:24:18.645764 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058645757, "job": 685, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21064}
>2018-07-11 04:24:18.645767 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 685] Level-0 flush table #10155: started
>2018-07-11 04:24:18.687850 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058687838, "cf_name": "default", "job": 685, "event": "table_file_creation", "file_number": 10155, "file_size": 19819, "table_properties": {"data_size": 18749, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18526, "raw_average_value_size": 2315, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:24:18.687860 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 685] Level-0 flush table #10155: 19819 bytes OK
>2018-07-11 04:24:18.737885 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.704428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10155 started
>2018-07-11 04:24:18.737894 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.737851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10155: memtable #1 done
>2018-07-11 04:24:18.737896 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.737866) EVENT_LOG_v1 {"time_micros": 1531283058737860, "job": 685, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:24:18.737898 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.737877) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:24:18.737905 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 685] Try to delete WAL files size 18700, prev total WAL file size 18700, number of live WAL files 2.
>
>2018-07-11 04:24:18.737996 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:24:18.738043 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.738037) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 04:24:18.738052 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 686] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:24:18.738057 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 685 Base level 0, inputs: [10155(19KB)], [10153(13MB)]
>
>2018-07-11 04:24:18.738070 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058738060, "job": 686, "event": "compaction_started", "files_L0": [10155], "files_L1": [10153], "score": -1, "input_data_size": 14606020}
>2018-07-11 04:24:18.923228 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 686] Generated table #10156: 3065 keys, 14604901 bytes
>2018-07-11 04:24:18.923260 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058923245, "cf_name": "default", "job": 686, "event": "table_file_creation", "file_number": 10156, "file_size": 14604901, "table_properties": {"data_size": 14491494, "index_size": 65032, "filter_size": 47448, "raw_key_size": 61768, "raw_average_key_size": 20, "raw_value_size": 14402202, "raw_average_value_size": 4698, "num_data_blocks": 1908, "num_entries": 3065, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:24:18.956372 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 686] Compacted 1@0 + 1@1 files to L1 => 14604901 bytes
>2018-07-11 04:24:18.989925 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.989859) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 66.9 rd, 66.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.9) out(13.9), read-write-amplify(1473.9) write-amplify(736.9) OK, records in: 3070, records dropped: 5
>
>2018-07-11 04:24:18.989934 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:24:18.989881) EVENT_LOG_v1 {"time_micros": 1531283058989872, "job": 686, "event": "compaction_finished", "compaction_time_micros": 218287, "output_level": 1, "num_output_files": 1, "total_output_size": 14604901, "num_input_records": 3070, "num_output_records": 3065, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:24:18.990013 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058990009, "job": 686, "event": "table_file_deletion", "file_number": 10155}
>2018-07-11 04:24:18.992694 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283058992691, "job": 686, "event": "table_file_deletion", "file_number": 10153}
>2018-07-11 04:24:23.160777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214872 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:24.207700 7f29149e4700  1 mon.magna102@0(leader).log v18517 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18517)
>2018-07-11 04:24:28.161055 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214875 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:28.274856 7f29149e4700  1 mon.magna102@0(leader).log v18518 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18518)
>2018-07-11 04:24:33.161289 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214878 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:34.217153 7f29149e4700  1 mon.magna102@0(leader).log v18519 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18519)
>2018-07-11 04:24:38.161475 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214881 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:38.276056 7f29149e4700  1 mon.magna102@0(leader).log v18520 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18520)
>2018-07-11 04:24:43.161704 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214884 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:44.226715 7f29149e4700  1 mon.magna102@0(leader).log v18521 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18521)
>2018-07-11 04:24:51.091726 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214887 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:51.244158 7f29149e4700  1 mon.magna102@0(leader).log v18522 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18522)
>2018-07-11 04:24:57.094484 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214890 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:24:57.244661 7f29149e4700  1 mon.magna102@0(leader).log v18523 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18523)
>2018-07-11 04:25:00.238943 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:25:03.097011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214893 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:03.245336 7f29149e4700  1 mon.magna102@0(leader).log v18524 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18524)
>2018-07-11 04:25:08.162608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214896 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:08.238654 7f29149e4700  1 mon.magna102@0(leader).log v18525 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18525)
>2018-07-11 04:25:17.103088 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214899 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:17.256346 7f29149e4700  1 mon.magna102@0(leader).log v18526 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18526)
>2018-07-11 04:25:23.105967 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214902 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:23.282904 7f29149e4700  1 mon.magna102@0(leader).log v18527 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18527)
>2018-07-11 04:25:28.163486 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214905 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:28.275021 7f29149e4700  1 mon.magna102@0(leader).log v18528 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18528)
>2018-07-11 04:25:33.163726 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214908 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:33.275694 7f29149e4700  1 mon.magna102@0(leader).log v18529 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18529)
>2018-07-11 04:25:43.114551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214911 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:43.259891 7f29149e4700  1 mon.magna102@0(leader).log v18530 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18530)
>2018-07-11 04:25:48.164327 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214914 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:48.252091 7f29149e4700  1 mon.magna102@0(leader).log v18531 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18531)
>2018-07-11 04:25:55.119587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214917 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:25:55.269562 7f29149e4700  1 mon.magna102@0(leader).log v18532 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18532)
>2018-07-11 04:26:00.239123 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:26:03.123248 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214923 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:03.271579 7f29149e4700  1 mon.magna102@0(leader).log v18533 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18533)
>2018-07-11 04:26:13.127389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214926 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:13.280766 7f29149e4700  1 mon.magna102@0(leader).log v18534 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18534)
>2018-07-11 04:26:18.165373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214929 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:18.264538 7f29149e4700  1 mon.magna102@0(leader).log v18535 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18535)
>2018-07-11 04:26:27.133477 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214932 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:27.290477 7f29149e4700  1 mon.magna102@0(leader).log v18536 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18536)
>2018-07-11 04:26:33.174076 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214935 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:33.291100 7f29149e4700  1 mon.magna102@0(leader).log v18537 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18537)
>2018-07-11 04:26:38.174290 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214938 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:38.274876 7f29149e4700  1 mon.magna102@0(leader).log v18538 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18538)
>2018-07-11 04:26:48.175334 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214941 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:48.275768 7f29149e4700  1 mon.magna102@0(leader).log v18539 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18539)
>2018-07-11 04:26:48.351292 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10157. Immutable memtables: 0.
>
>2018-07-11 04:26:48.351337 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 687] Syncing log #10154
>2018-07-11 04:26:48.367710 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.351327) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:26:48.367722 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 687] Flushing memtable with next log file: 10157
>
>2018-07-11 04:26:48.367748 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208367740, "job": 687, "event": "flush_started", "num_memtables": 1, "num_entries": 965, "num_deletes": 251, "memory_usage": 1577120}
>2018-07-11 04:26:48.367752 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 687] Level-0 flush table #10158: started
>2018-07-11 04:26:48.418313 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208418298, "cf_name": "default", "job": 687, "event": "table_file_creation", "file_number": 10158, "file_size": 1518699, "table_properties": {"data_size": 1505820, "index_size": 6545, "filter_size": 5414, "raw_key_size": 9800, "raw_average_key_size": 20, "raw_value_size": 1494495, "raw_average_value_size": 3120, "num_data_blocks": 199, "num_entries": 479, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:26:48.418324 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 687] Level-0 flush table #10158: 1518699 bytes OK
>2018-07-11 04:26:48.468283 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.434819) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10158 started
>2018-07-11 04:26:48.468291 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.468247) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10158: memtable #1 done
>2018-07-11 04:26:48.468294 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.468262) EVENT_LOG_v1 {"time_micros": 1531283208468255, "job": 687, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:26:48.468309 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.468274) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:26:48.468315 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 687] Try to delete WAL files size 1556800, prev total WAL file size 1556800, number of live WAL files 2.
>
>2018-07-11 04:26:48.468782 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:26:48.468848 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.468841) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:26:48.468855 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 688] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:26:48.468862 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 687 Base level 0, inputs: [10158(1483KB)], [10156(13MB)]
>
>2018-07-11 04:26:48.468876 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208468866, "job": 688, "event": "compaction_started", "files_L0": [10158], "files_L1": [10156], "score": -1, "input_data_size": 16123600}
>2018-07-11 04:26:48.645280 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 688] Generated table #10159: 3034 keys, 14138970 bytes
>2018-07-11 04:26:48.645312 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208645297, "cf_name": "default", "job": 688, "event": "table_file_creation", "file_number": 10159, "file_size": 14138970, "table_properties": {"data_size": 14028382, "index_size": 63538, "filter_size": 46123, "raw_key_size": 61321, "raw_average_key_size": 20, "raw_value_size": 13940506, "raw_average_value_size": 4594, "num_data_blocks": 1856, "num_entries": 3034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:26:48.686769 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 688] Compacted 1@0 + 1@1 files to L1 => 14138970 bytes
>2018-07-11 04:26:48.728632 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.728567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.0 rd, 64.9 wr, level 1, files in(1, 1) out(1) MB in(1.4, 13.9) out(13.5), read-write-amplify(19.9) write-amplify(9.3) OK, records in: 3544, records dropped: 510
>
>2018-07-11 04:26:48.728641 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:26:48.728588) EVENT_LOG_v1 {"time_micros": 1531283208728579, "job": 688, "event": "compaction_finished", "compaction_time_micros": 217877, "output_level": 1, "num_output_files": 1, "total_output_size": 14138970, "num_input_records": 3544, "num_output_records": 3034, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:26:48.729033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208729029, "job": 688, "event": "table_file_deletion", "file_number": 10158}
>2018-07-11 04:26:48.731894 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283208731891, "job": 688, "event": "table_file_deletion", "file_number": 10156}
>2018-07-11 04:26:53.179350 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214944 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:53.296417 7f29149e4700  1 mon.magna102@0(leader).log v18540 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18540)
>2018-07-11 04:26:58.179569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214947 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:26:58.271865 7f29149e4700  1 mon.magna102@0(leader).log v18541 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18541)
>2018-07-11 04:27:00.239302 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:27:07.151098 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214953 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:07.306216 7f29149e4700  1 mon.magna102@0(leader).log v18542 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18542)
>2018-07-11 04:27:15.154350 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214956 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:15.306934 7f29149e4700  1 mon.magna102@0(leader).log v18543 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18543)
>2018-07-11 04:27:23.190689 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214959 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:23.307624 7f29149e4700  1 mon.magna102@0(leader).log v18544 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18544)
>2018-07-11 04:27:28.190869 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214962 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:28.283244 7f29149e4700  1 mon.magna102@0(leader).log v18545 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18545)
>2018-07-11 04:27:33.191866 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214965 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:33.308908 7f29149e4700  1 mon.magna102@0(leader).log v18546 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18546)
>2018-07-11 04:27:39.164886 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214968 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:39.319084 7f29149e4700  1 mon.magna102@0(leader).log v18547 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18547)
>2018-07-11 04:27:45.167171 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214971 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:45.319864 7f29149e4700  1 mon.magna102@0(leader).log v18548 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18548)
>2018-07-11 04:27:51.169879 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214974 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:51.320457 7f29149e4700  1 mon.magna102@0(leader).log v18549 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18549)
>2018-07-11 04:27:59.173405 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214977 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:27:59.329519 7f29149e4700  1 mon.magna102@0(leader).log v18550 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18550)
>2018-07-11 04:28:00.239480 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:28:07.176887 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214980 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:07.330235 7f29149e4700  1 mon.magna102@0(leader).log v18551 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18551)
>2018-07-11 04:28:13.213809 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214983 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:13.347531 7f29149e4700  1 mon.magna102@0(leader).log v18552 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18552)
>2018-07-11 04:28:18.213983 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214986 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:18.422959 7f29149e4700  1 mon.magna102@0(leader).log v18553 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18553)
>2018-07-11 04:28:25.184571 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214989 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:25.357378 7f29149e4700  1 mon.magna102@0(leader).log v18554 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18554)
>2018-07-11 04:28:28.307806 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10160. Immutable memtables: 0.
>
>2018-07-11 04:28:28.307851 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 689] Syncing log #10157
>2018-07-11 04:28:28.324329 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.307840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:28:28.324340 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 689] Flushing memtable with next log file: 10160
>
>2018-07-11 04:28:28.324367 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308324359, "job": 689, "event": "flush_started", "num_memtables": 1, "num_entries": 725, "num_deletes": 250, "memory_usage": 1053064}
>2018-07-11 04:28:28.324370 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 689] Level-0 flush table #10161: started
>2018-07-11 04:28:28.374897 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308374883, "cf_name": "default", "job": 689, "event": "table_file_creation", "file_number": 10161, "file_size": 711698, "table_properties": {"data_size": 705099, "index_size": 2843, "filter_size": 2837, "raw_key_size": 7776, "raw_average_key_size": 21, "raw_value_size": 698120, "raw_average_value_size": 1939, "num_data_blocks": 88, "num_entries": 360, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:28:28.374908 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 689] Level-0 flush table #10161: 711698 bytes OK
>2018-07-11 04:28:28.433270 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.391443) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10161 started
>2018-07-11 04:28:28.433278 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.433234) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10161: memtable #1 done
>2018-07-11 04:28:28.433281 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.433249) EVENT_LOG_v1 {"time_micros": 1531283308433243, "job": 689, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:28:28.433296 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.433261) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:28:28.433302 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 689] Try to delete WAL files size 1037393, prev total WAL file size 1037393, number of live WAL files 2.
>
>2018-07-11 04:28:28.433636 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:28:28.433687 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.433680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:28:28.433697 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 690] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:28:28.433703 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 689 Base level 0, inputs: [10161(695KB)], [10159(13MB)]
>
>2018-07-11 04:28:28.433719 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308433706, "job": 690, "event": "compaction_started", "files_L0": [10161], "files_L1": [10159], "score": -1, "input_data_size": 14850668}
>2018-07-11 04:28:28.601708 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 690] Generated table #10162: 2930 keys, 13439509 bytes
>2018-07-11 04:28:28.601750 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308601724, "cf_name": "default", "job": 690, "event": "table_file_creation", "file_number": 10162, "file_size": 13439509, "table_properties": {"data_size": 13335583, "index_size": 59267, "filter_size": 43732, "raw_key_size": 58856, "raw_average_key_size": 20, "raw_value_size": 13252255, "raw_average_value_size": 4522, "num_data_blocks": 1737, "num_entries": 2930, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:28:28.635027 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 690] Compacted 1@0 + 1@1 files to L1 => 13439509 bytes
>2018-07-11 04:28:28.668624 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.668547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.8 rd, 66.8 wr, level 1, files in(1, 1) out(1) MB in(0.7, 13.5) out(12.8), read-write-amplify(39.8) write-amplify(18.9) OK, records in: 3394, records dropped: 464
>
>2018-07-11 04:28:28.668634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:28:28.668571) EVENT_LOG_v1 {"time_micros": 1531283308668561, "job": 690, "event": "compaction_finished", "compaction_time_micros": 201292, "output_level": 1, "num_output_files": 1, "total_output_size": 13439509, "num_input_records": 3394, "num_output_records": 2930, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:28:28.668848 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308668844, "job": 690, "event": "table_file_deletion", "file_number": 10161}
>2018-07-11 04:28:28.671520 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283308671517, "job": 690, "event": "table_file_deletion", "file_number": 10159}
>2018-07-11 04:28:28.671635 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:28:33.266076 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214992 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:33.394714 7f29149e4700  1 mon.magna102@0(leader).log v18555 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18555)
>2018-07-11 04:28:38.266280 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24214995 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:38.353485 7f29149e4700  1 mon.magna102@0(leader).log v18556 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18556)
>2018-07-11 04:28:47.194396 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215001 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:47.371038 7f29149e4700  1 mon.magna102@0(leader).log v18557 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18557)
>2018-07-11 04:28:55.197659 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215004 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:28:55.347005 7f29149e4700  1 mon.magna102@0(leader).log v18558 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18558)
>2018-07-11 04:29:00.239658 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:29:03.267129 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215007 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:04.331059 7f29149e4700  1 mon.magna102@0(leader).log v18559 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18559)
>2018-07-11 04:29:08.267331 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215010 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:08.339873 7f29149e4700  1 mon.magna102@0(leader).log v18560 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18560)
>2018-07-11 04:29:13.267518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215013 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:14.332164 7f29149e4700  1 mon.magna102@0(leader).log v18561 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18561)
>2018-07-11 04:29:19.208027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215016 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:19.359028 7f29149e4700  1 mon.magna102@0(leader).log v18562 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18562)
>2018-07-11 04:29:25.210467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215019 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:25.359604 7f29149e4700  1 mon.magna102@0(leader).log v18563 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18563)
>2018-07-11 04:29:33.214017 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215022 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:33.368652 7f29149e4700  1 mon.magna102@0(leader).log v18564 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18564)
>2018-07-11 04:29:38.268416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215025 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:38.352568 7f29149e4700  1 mon.magna102@0(leader).log v18565 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18565)
>2018-07-11 04:29:47.220153 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215028 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:47.395142 7f29149e4700  1 mon.magna102@0(leader).log v18566 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18566)
>2018-07-11 04:29:55.223438 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215031 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:29:55.379313 7f29149e4700  1 mon.magna102@0(leader).log v18567 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18567)
>2018-07-11 04:30:00.239835 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:30:03.271397 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215034 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:03.388494 7f29149e4700  1 mon.magna102@0(leader).log v18568 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18568)
>2018-07-11 04:30:08.271679 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215037 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:08.372282 7f29149e4700  1 mon.magna102@0(leader).log v18569 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18569)
>2018-07-11 04:30:18.272711 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215040 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:18.399395 7f29149e4700  1 mon.magna102@0(leader).log v18570 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18570)
>2018-07-11 04:30:23.272935 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215046 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:23.400048 7f29149e4700  1 mon.magna102@0(leader).log v18571 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18571)
>2018-07-11 04:30:25.158504 7f29149e4700  1 mon.magna102@0(leader).log v18572 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18572)
>2018-07-11 04:30:27.292195 7f29149e4700  1 mon.magna102@0(leader).log v18573 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18573)
>2018-07-11 04:30:31.239236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215049 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:31.384548 7f29149e4700  1 mon.magna102@0(leader).log v18574 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18574)
>2018-07-11 04:30:39.242608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215052 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:39.402083 7f29149e4700  1 mon.magna102@0(leader).log v18575 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18575)
>2018-07-11 04:30:48.285724 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215055 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:48.377991 7f29149e4700  1 mon.magna102@0(leader).log v18576 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18576)
>2018-07-11 04:30:52.461759 7f29149e4700  1 mon.magna102@0(leader).log v18577 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18577)
>2018-07-11 04:30:53.285964 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215058 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:53.528609 7f29149e4700  1 mon.magna102@0(leader).log v18578 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18578)
>2018-07-11 04:30:56.570742 7f29149e4700  1 mon.magna102@0(leader).log v18579 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18579)
>2018-07-11 04:30:58.286164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215061 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:30:58.696122 7f29149e4700  1 mon.magna102@0(leader).log v18580 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18580)
>2018-07-11 04:31:00.240022 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:31:02.454899 7f29149e4700  1 mon.magna102@0(leader).log v18581 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18581)
>2018-07-11 04:31:03.454940 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215064 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:03.605322 7f29149e4700  1 mon.magna102@0(leader).log v18582 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18582)
>2018-07-11 04:31:13.257350 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215070 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:13.415736 7f29149e4700  1 mon.magna102@0(leader).log v18583 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18583)
>2018-07-11 04:31:19.259901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215073 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:19.416335 7f29149e4700  1 mon.magna102@0(leader).log v18584 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18584)
>2018-07-11 04:31:27.263422 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215076 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:27.417069 7f29149e4700  1 mon.magna102@0(leader).log v18585 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18585)
>2018-07-11 04:31:33.265853 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215079 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:33.426075 7f29149e4700  1 mon.magna102@0(leader).log v18586 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18586)
>2018-07-11 04:31:39.268400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215082 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:39.443352 7f29149e4700  1 mon.magna102@0(leader).log v18587 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18587)
>2018-07-11 04:31:48.310261 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215085 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:48.435734 7f29149e4700  1 mon.magna102@0(leader).log v18588 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18588)
>2018-07-11 04:31:53.311021 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215088 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:53.444714 7f29149e4700  1 mon.magna102@0(leader).log v18589 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18589)
>2018-07-11 04:31:59.276986 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215091 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:31:59.453737 7f29149e4700  1 mon.magna102@0(leader).log v18590 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18590)
>2018-07-11 04:32:00.240218 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:32:08.312399 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215094 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:08.405671 7f29149e4700  1 mon.magna102@0(leader).log v18591 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18591)
>2018-07-11 04:32:13.314328 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215097 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:13.431330 7f29149e4700  1 mon.magna102@0(leader).log v18592 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18592)
>2018-07-11 04:32:18.314491 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215100 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:18.415139 7f29149e4700  1 mon.magna102@0(leader).log v18593 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18593)
>2018-07-11 04:32:28.323886 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215103 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:28.424346 7f29149e4700  1 mon.magna102@0(leader).log v18594 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18594)
>2018-07-11 04:32:33.332932 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215106 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:33.450045 7f29149e4700  1 mon.magna102@0(leader).log v18595 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18595)
>2018-07-11 04:32:38.333142 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215109 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:38.425480 7f29149e4700  1 mon.magna102@0(leader).log v18596 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18596)
>2018-07-11 04:32:43.333386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215112 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:43.451144 7f29149e4700  1 mon.magna102@0(leader).log v18597 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18597)
>2018-07-11 04:32:48.333631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215115 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:48.434932 7f29149e4700  1 mon.magna102@0(leader).log v18598 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18598)
>2018-07-11 04:32:58.335395 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215118 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:32:58.444147 7f29149e4700  1 mon.magna102@0(leader).log v18599 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18599)
>2018-07-11 04:33:00.240399 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:33:03.345685 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215121 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:03.479387 7f29149e4700  1 mon.magna102@0(leader).log v18600 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18600)
>2018-07-11 04:33:03.563114 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10163. Immutable memtables: 0.
>
>2018-07-11 04:33:03.563157 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 691] Syncing log #10160
>2018-07-11 04:33:03.579682 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.563148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:33:03.579694 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 691] Flushing memtable with next log file: 10163
>
>2018-07-11 04:33:03.579707 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583579700, "job": 691, "event": "flush_started", "num_memtables": 1, "num_entries": 1590, "num_deletes": 251, "memory_usage": 2928328}
>2018-07-11 04:33:03.579710 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 691] Level-0 flush table #10164: started
>2018-07-11 04:33:03.646927 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583646912, "cf_name": "default", "job": 691, "event": "table_file_creation", "file_number": 10164, "file_size": 2824449, "table_properties": {"data_size": 2801631, "index_size": 12345, "filter_size": 9549, "raw_key_size": 13849, "raw_average_key_size": 20, "raw_value_size": 2783410, "raw_average_value_size": 4129, "num_data_blocks": 372, "num_entries": 674, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:33:03.646938 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 691] Level-0 flush table #10164: 2824449 bytes OK
>2018-07-11 04:33:03.696949 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.663455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10164 started
>2018-07-11 04:33:03.696958 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.696912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10164: memtable #1 done
>2018-07-11 04:33:03.696961 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.696927) EVENT_LOG_v1 {"time_micros": 1531283583696921, "job": 691, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:33:03.696963 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.696939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:33:03.696970 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 691] Try to delete WAL files size 2896245, prev total WAL file size 2896245, number of live WAL files 2.
>
>2018-07-11 04:33:03.697709 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:33:03.697929 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 04:33:03.697938 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 45610.6 total, 775.4 interval
>Cumulative writes: 64K writes, 308K keys, 64K commit groups, 1.0 writes per commit group, ingest: 0.52 GB, 0.01 MB/s
>Cumulative WAL: 64K writes, 64K syncs, 1.00 writes per sync, written: 0.52 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1032 writes, 5228 keys, 1032 commit groups, 1.0 writes per commit group, ingest: 7.73 MB, 0.01 MB/s
>Interval WAL: 1032 writes, 1032 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.69 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        29       346    0.083       0      0
>  L1      1/1   12.82 MB   0.0      5.4     0.5      5.0       5.0     -0.0       0.0  10.9     72.3     66.2        77       344    0.223   1216K   143K
> Sum      2/2   15.51 MB   0.0      5.4     0.5      5.0       5.4      0.5       0.0  11.9     52.6     52.6       105       690    0.153   1216K   143K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  14.6     52.0     51.7         2        14    0.147     25K   2837
>Uptime(secs): 45610.6 total, 45610.6 interval
>Flush(GB): cumulative 0.456, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.41 GB write, 0.12 MB/s write, 5.41 GB read, 0.12 MB/s read, 105.5 seconds
>Interval compaction: 0.10 GB write, 0.00 MB/s write, 0.10 GB read, 0.00 MB/s read, 2.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.69 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        29       346    0.083       0      0
>  L1      1/1   12.82 MB   0.0      5.4     0.5      5.0       5.0     -0.0       0.0  10.9     72.3     66.2        77       344    0.223   1216K   143K
> Sum      2/2   15.51 MB   0.0      5.4     0.5      5.0       5.4      0.5       0.0  11.9     52.6     52.6       105       690    0.153   1216K   143K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 45610.6 total, 45610.6 interval
>Flush(GB): cumulative 0.456, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.41 GB write, 0.12 MB/s write, 5.41 GB read, 0.12 MB/s read, 105.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 04:33:03.697961 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.697953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:33:03.697966 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 692] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:33:03.697971 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 691 Base level 0, inputs: [10164(2758KB)], [10162(12MB)]
>
>2018-07-11 04:33:03.697985 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583697975, "job": 692, "event": "compaction_started", "files_L0": [10164], "files_L1": [10162], "score": -1, "input_data_size": 16263958}
>2018-07-11 04:33:03.872668 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 692] Generated table #10165: 3094 keys, 14278610 bytes
>2018-07-11 04:33:03.872712 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583872685, "cf_name": "default", "job": 692, "event": "table_file_creation", "file_number": 10165, "file_size": 14278610, "table_properties": {"data_size": 14167445, "index_size": 63687, "filter_size": 46551, "raw_key_size": 62458, "raw_average_key_size": 20, "raw_value_size": 14078622, "raw_average_value_size": 4550, "num_data_blocks": 1859, "num_entries": 3094, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:33:03.905866 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 692] Compacted 1@0 + 1@1 files to L1 => 14278610 bytes
>2018-07-11 04:33:03.939369 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.939302) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 78.2 rd, 68.7 wr, level 1, files in(1, 1) out(1) MB in(2.7, 12.8) out(13.6), read-write-amplify(10.8) write-amplify(5.1) OK, records in: 3604, records dropped: 510
>
>2018-07-11 04:33:03.939378 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:33:03.939324) EVENT_LOG_v1 {"time_micros": 1531283583939314, "job": 692, "event": "compaction_finished", "compaction_time_micros": 207865, "output_level": 1, "num_output_files": 1, "total_output_size": 14278610, "num_input_records": 3604, "num_output_records": 3094, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:33:03.939974 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583939970, "job": 692, "event": "table_file_deletion", "file_number": 10164}
>2018-07-11 04:33:03.942613 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283583942610, "job": 692, "event": "table_file_deletion", "file_number": 10162}
>2018-07-11 04:33:08.345890 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215124 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:08.448861 7f29149e4700  1 mon.magna102@0(leader).log v18601 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18601)
>2018-07-11 04:33:18.349304 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215127 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:18.483029 7f29149e4700  1 mon.magna102@0(leader).log v18602 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18602)
>2018-07-11 04:33:23.350044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215130 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:23.483743 7f29149e4700  1 mon.magna102@0(leader).log v18603 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18603)
>2018-07-11 04:33:28.350257 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215133 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:28.475849 7f29149e4700  1 mon.magna102@0(leader).log v18604 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18604)
>2018-07-11 04:33:33.359427 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215136 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:33.493204 7f29149e4700  1 mon.magna102@0(leader).log v18605 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18605)
>2018-07-11 04:33:38.359656 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215139 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:38.460309 7f29149e4700  1 mon.magna102@0(leader).log v18606 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18606)
>2018-07-11 04:33:43.361872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215142 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:43.478965 7f29149e4700  1 mon.magna102@0(leader).log v18607 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18607)
>2018-07-11 04:33:48.362106 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215145 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:48.454401 7f29149e4700  1 mon.magna102@0(leader).log v18608 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18608)
>2018-07-11 04:33:58.363211 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215148 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:33:58.463732 7f29149e4700  1 mon.magna102@0(leader).log v18609 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18609)
>2018-07-11 04:34:00.240584 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:34:08.364221 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215151 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:08.472960 7f29149e4700  1 mon.magna102@0(leader).log v18610 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18610)
>2018-07-11 04:34:13.364896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215154 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:13.481957 7f29149e4700  1 mon.magna102@0(leader).log v18611 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18611)
>2018-07-11 04:34:18.365089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215157 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:18.482318 7f29149e4700  1 mon.magna102@0(leader).log v18612 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18612)
>2018-07-11 04:34:28.374534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215160 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:28.466494 7f29149e4700  1 mon.magna102@0(leader).log v18613 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18613)
>2018-07-11 04:34:33.375232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215163 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:33.492271 7f29149e4700  1 mon.magna102@0(leader).log v18614 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18614)
>2018-07-11 04:34:38.375442 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215166 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:38.485632 7f29149e4700  1 mon.magna102@0(leader).log v18615 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18615)
>2018-07-11 04:34:43.394279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215169 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:43.527982 7f29149e4700  1 mon.magna102@0(leader).log v18616 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18616)
>2018-07-11 04:34:49.350021 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215172 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:49.520202 7f29149e4700  1 mon.magna102@0(leader).log v18617 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18617)
>2018-07-11 04:34:58.395554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215175 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:34:58.521046 7f29149e4700  1 mon.magna102@0(leader).log v18618 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18618)
>2018-07-11 04:35:00.240768 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:35:03.395782 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215178 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:03.530041 7f29149e4700  1 mon.magna102@0(leader).log v18619 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18619)
>2018-07-11 04:35:08.396011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215181 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:08.497157 7f29149e4700  1 mon.magna102@0(leader).log v18620 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18620)
>2018-07-11 04:35:18.396380 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215184 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:18.498064 7f29149e4700  1 mon.magna102@0(leader).log v18621 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18621)
>2018-07-11 04:35:23.398343 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215187 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:23.515367 7f29149e4700  1 mon.magna102@0(leader).log v18622 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18622)
>2018-07-11 04:35:28.398550 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215190 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:28.499158 7f29149e4700  1 mon.magna102@0(leader).log v18623 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18623)
>2018-07-11 04:35:33.407749 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215193 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:33.524856 7f29149e4700  1 mon.magna102@0(leader).log v18624 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18624)
>2018-07-11 04:35:38.407955 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215196 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:38.501605 7f29149e4700  1 mon.magna102@0(leader).log v18625 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18625)
>2018-07-11 04:35:48.410419 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215199 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:48.510849 7f29149e4700  1 mon.magna102@0(leader).log v18626 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18626)
>2018-07-11 04:35:53.419419 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215202 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:53.536500 7f29149e4700  1 mon.magna102@0(leader).log v18627 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18627)
>2018-07-11 04:35:58.419642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215205 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:35:58.511952 7f29149e4700  1 mon.magna102@0(leader).log v18628 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18628)
>2018-07-11 04:36:00.240943 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:36:07.383194 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215211 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:07.537851 7f29149e4700  1 mon.magna102@0(leader).log v18629 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18629)
>2018-07-11 04:36:13.385632 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215214 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:13.530314 7f29149e4700  1 mon.magna102@0(leader).log v18630 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18630)
>2018-07-11 04:36:21.389041 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215217 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:21.539398 7f29149e4700  1 mon.magna102@0(leader).log v18631 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18631)
>2018-07-11 04:36:29.392412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215220 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:29.565064 7f29149e4700  1 mon.magna102@0(leader).log v18632 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18632)
>2018-07-11 04:36:35.395065 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215223 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:35.550260 7f29149e4700  1 mon.magna102@0(leader).log v18633 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18633)
>2018-07-11 04:36:43.434015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215226 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:43.551015 7f29149e4700  1 mon.magna102@0(leader).log v18634 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18634)
>2018-07-11 04:36:48.434232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215229 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:48.510060 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10166. Immutable memtables: 0.
>
>2018-07-11 04:36:48.510104 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 693] Syncing log #10163
>2018-07-11 04:36:48.526556 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.510094) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:36:48.526567 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 693] Flushing memtable with next log file: 10166
>
>2018-07-11 04:36:48.526593 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808526585, "job": 693, "event": "flush_started", "num_memtables": 1, "num_entries": 1311, "num_deletes": 250, "memory_usage": 2312056}
>2018-07-11 04:36:48.526596 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 693] Level-0 flush table #10167: started
>2018-07-11 04:36:48.610640 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808610626, "cf_name": "default", "job": 693, "event": "table_file_creation", "file_number": 10167, "file_size": 1503702, "table_properties": {"data_size": 1491340, "index_size": 6120, "filter_size": 5322, "raw_key_size": 10175, "raw_average_key_size": 21, "raw_value_size": 1480322, "raw_average_value_size": 3090, "num_data_blocks": 189, "num_entries": 479, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:36:48.610651 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 693] Level-0 flush table #10167: 1503702 bytes OK
>2018-07-11 04:36:48.644055 7f29149e4700  1 mon.magna102@0(leader).log v18635 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18635)
>2018-07-11 04:36:48.685535 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.643818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10167 started
>2018-07-11 04:36:48.685543 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.685499) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10167: memtable #1 done
>2018-07-11 04:36:48.685546 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.685514) EVENT_LOG_v1 {"time_micros": 1531283808685507, "job": 693, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:36:48.685547 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.685525) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:36:48.685566 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 693] Try to delete WAL files size 2285649, prev total WAL file size 2304349, number of live WAL files 2.
>
>2018-07-11 04:36:48.686169 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:36:48.686243 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.686236) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:36:48.686253 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 694] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:36:48.686260 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 693 Base level 0, inputs: [10167(1468KB)], [10165(13MB)]
>
>2018-07-11 04:36:48.686273 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808686263, "job": 694, "event": "compaction_started", "files_L0": [10167], "files_L1": [10165], "score": -1, "input_data_size": 15782312}
>2018-07-11 04:36:48.861441 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 694] Generated table #10168: 3171 keys, 14794465 bytes
>2018-07-11 04:36:48.861494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808861478, "cf_name": "default", "job": 694, "event": "table_file_creation", "file_number": 10168, "file_size": 14794465, "table_properties": {"data_size": 14680560, "index_size": 64956, "filter_size": 48022, "raw_key_size": 63756, "raw_average_key_size": 20, "raw_value_size": 14589841, "raw_average_value_size": 4601, "num_data_blocks": 1903, "num_entries": 3171, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:36:48.894627 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 694] Compacted 1@0 + 1@1 files to L1 => 14794465 bytes
>2018-07-11 04:36:48.936417 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.936351) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.8 rd, 71.0 wr, level 1, files in(1, 1) out(1) MB in(1.4, 13.6) out(14.1), read-write-amplify(20.3) write-amplify(9.8) OK, records in: 3573, records dropped: 402
>
>2018-07-11 04:36:48.936426 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.936372) EVENT_LOG_v1 {"time_micros": 1531283808936363, "job": 694, "event": "compaction_finished", "compaction_time_micros": 208337, "output_level": 1, "num_output_files": 1, "total_output_size": 14794465, "num_input_records": 3573, "num_output_records": 3171, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:36:48.936812 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808936808, "job": 694, "event": "table_file_deletion", "file_number": 10167}
>2018-07-11 04:36:48.939465 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808939462, "job": 694, "event": "table_file_deletion", "file_number": 10165}
>2018-07-11 04:36:48.939784 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10169. Immutable memtables: 0.
>
>2018-07-11 04:36:48.939822 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 695] Syncing log #10166
>2018-07-11 04:36:48.953359 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:48.939814) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:36:48.953371 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 695] Flushing memtable with next log file: 10169
>
>2018-07-11 04:36:48.953397 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808953390, "job": 695, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21056}
>2018-07-11 04:36:48.953400 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 695] Level-0 flush table #10170: started
>2018-07-11 04:36:48.995491 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283808995478, "cf_name": "default", "job": 695, "event": "table_file_creation", "file_number": 10170, "file_size": 19819, "table_properties": {"data_size": 18749, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18526, "raw_average_value_size": 2315, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:36:48.995501 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 695] Level-0 flush table #10170: 19819 bytes OK
>2018-07-11 04:36:49.045560 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.012036) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10170 started
>2018-07-11 04:36:49.045568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.045526) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10170: memtable #1 done
>2018-07-11 04:36:49.045571 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.045541) EVENT_LOG_v1 {"time_micros": 1531283809045534, "job": 695, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:36:49.045585 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.045552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:36:49.045591 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 695] Try to delete WAL files size 18700, prev total WAL file size 18700, number of live WAL files 2.
>
>2018-07-11 04:36:49.045688 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:36:49.045734 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.045728) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:36:49.045744 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 696] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:36:49.045750 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 695 Base level 0, inputs: [10170(19KB)], [10168(14MB)]
>
>2018-07-11 04:36:49.045763 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283809045753, "job": 696, "event": "compaction_started", "files_L0": [10170], "files_L1": [10168], "score": -1, "input_data_size": 14814284}
>2018-07-11 04:36:49.221299 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 696] Generated table #10171: 3174 keys, 14813181 bytes
>2018-07-11 04:36:49.221343 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283809221327, "cf_name": "default", "job": 696, "event": "table_file_creation", "file_number": 10171, "file_size": 14813181, "table_properties": {"data_size": 14699150, "index_size": 65025, "filter_size": 48079, "raw_key_size": 63817, "raw_average_key_size": 20, "raw_value_size": 14608327, "raw_average_value_size": 4602, "num_data_blocks": 1905, "num_entries": 3174, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:36:49.262836 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 696] Compacted 1@0 + 1@1 files to L1 => 14813181 bytes
>2018-07-11 04:36:49.296351 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.296280) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.3 rd, 68.2 wr, level 1, files in(1, 1) out(1) MB in(0.0, 14.1) out(14.1), read-write-amplify(1494.9) write-amplify(747.4) OK, records in: 3179, records dropped: 5
>
>2018-07-11 04:36:49.296360 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:36:49.296302) EVENT_LOG_v1 {"time_micros": 1531283809296292, "job": 696, "event": "compaction_finished", "compaction_time_micros": 217059, "output_level": 1, "num_output_files": 1, "total_output_size": 14813181, "num_input_records": 3179, "num_output_records": 3174, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:36:49.296455 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283809296450, "job": 696, "event": "table_file_deletion", "file_number": 10170}
>2018-07-11 04:36:49.299387 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283809299384, "job": 696, "event": "table_file_deletion", "file_number": 10168}
>2018-07-11 04:36:58.476755 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215232 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:36:58.597717 7f29149e4700  1 mon.magna102@0(leader).log v18636 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18636)
>2018-07-11 04:37:00.241124 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:37:03.477004 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215235 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:03.591292 7f29149e4700  1 mon.magna102@0(leader).log v18637 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18637)
>2018-07-11 04:37:08.477230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215238 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:08.591706 7f29149e4700  1 mon.magna102@0(leader).log v18638 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18638)
>2018-07-11 04:37:13.477457 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215241 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:14.550684 7f29149e4700  1 mon.magna102@0(leader).log v18639 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18639)
>2018-07-11 04:37:19.414169 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215244 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:19.567977 7f29149e4700  1 mon.magna102@0(leader).log v18640 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18640)
>2018-07-11 04:37:27.417548 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215250 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:27.568804 7f29149e4700  1 mon.magna102@0(leader).log v18641 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18641)
>2018-07-11 04:37:35.421013 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215253 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:35.577875 7f29149e4700  1 mon.magna102@0(leader).log v18642 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18642)
>2018-07-11 04:37:41.423539 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215256 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:41.578457 7f29149e4700  1 mon.magna102@0(leader).log v18643 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18643)
>2018-07-11 04:37:49.427089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215259 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:49.587518 7f29149e4700  1 mon.magna102@0(leader).log v18644 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18644)
>2018-07-11 04:37:58.479019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215262 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:37:58.596708 7f29149e4700  1 mon.magna102@0(leader).log v18645 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18645)
>2018-07-11 04:38:00.241313 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:38:03.479248 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215265 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:03.598597 7f29149e4700  1 mon.magna102@0(leader).log v18646 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18646)
>2018-07-11 04:38:08.479481 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215268 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:08.565670 7f29149e4700  1 mon.magna102@0(leader).log v18647 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18647)
>2018-07-11 04:38:15.438190 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215271 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:15.591469 7f29149e4700  1 mon.magna102@0(leader).log v18648 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18648)
>2018-07-11 04:38:23.441663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215277 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:23.592340 7f29149e4700  1 mon.magna102@0(leader).log v18649 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18649)
>2018-07-11 04:38:33.446065 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215280 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:33.618387 7f29149e4700  1 mon.magna102@0(leader).log v18650 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18650)
>2018-07-11 04:38:38.493148 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215283 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:38.577132 7f29149e4700  1 mon.magna102@0(leader).log v18651 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18651)
>2018-07-11 04:38:48.493514 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215286 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:48.594705 7f29149e4700  1 mon.magna102@0(leader).log v18652 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18652)
>2018-07-11 04:38:55.455329 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215292 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:38:55.603916 7f29149e4700  1 mon.magna102@0(leader).log v18653 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18653)
>2018-07-11 04:39:00.241501 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:39:01.457898 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215295 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:01.614091 7f29149e4700  1 mon.magna102@0(leader).log v18654 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18654)
>2018-07-11 04:39:07.460423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215298 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:07.614674 7f29149e4700  1 mon.magna102@0(leader).log v18655 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18655)
>2018-07-11 04:39:13.463163 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215301 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:13.623501 7f29149e4700  1 mon.magna102@0(leader).log v18656 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18656)
>2018-07-11 04:39:23.467359 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215304 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:23.641155 7f29149e4700  1 mon.magna102@0(leader).log v18657 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18657)
>2018-07-11 04:39:27.633455 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10172. Immutable memtables: 0.
>
>2018-07-11 04:39:27.633499 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 697] Syncing log #10169
>2018-07-11 04:39:27.649978 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.633489) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:39:27.649990 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 697] Flushing memtable with next log file: 10172
>
>2018-07-11 04:39:27.650004 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283967649997, "job": 697, "event": "flush_started", "num_memtables": 1, "num_entries": 992, "num_deletes": 251, "memory_usage": 1633240}
>2018-07-11 04:39:27.650007 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 697] Level-0 flush table #10173: started
>2018-07-11 04:39:27.700568 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283967700553, "cf_name": "default", "job": 697, "event": "table_file_creation", "file_number": 10173, "file_size": 1571502, "table_properties": {"data_size": 1558193, "index_size": 6805, "filter_size": 5584, "raw_key_size": 9949, "raw_average_key_size": 20, "raw_value_size": 1546578, "raw_average_value_size": 3182, "num_data_blocks": 207, "num_entries": 486, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:39:27.700578 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 697] Level-0 flush table #10173: 1571502 bytes OK
>2018-07-11 04:39:27.750539 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.717130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10173 started
>2018-07-11 04:39:27.750548 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.750501) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10173: memtable #1 done
>2018-07-11 04:39:27.750550 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.750517) EVENT_LOG_v1 {"time_micros": 1531283967750510, "job": 697, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:39:27.750552 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.750529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:39:27.750559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 697] Try to delete WAL files size 1612289, prev total WAL file size 1612289, number of live WAL files 2.
>
>2018-07-11 04:39:27.751015 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:39:27.751069 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:39:27.751062) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:39:27.751079 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 698] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:39:27.751085 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 697 Base level 0, inputs: [10173(1534KB)], [10171(14MB)]
>
>2018-07-11 04:39:27.751098 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283967751088, "job": 698, "event": "compaction_started", "files_L0": [10173], "files_L1": [10171], "score": -1, "input_data_size": 16384683}
>2018-07-11 04:39:27.942915 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 698] Generated table #10174: 3150 keys, 14375354 bytes
>2018-07-11 04:39:27.942970 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283967942949, "cf_name": "default", "job": 698, "event": "table_file_creation", "file_number": 10174, "file_size": 14375354, "table_properties": {"data_size": 14263791, "index_size": 63767, "filter_size": 46869, "raw_key_size": 63519, "raw_average_key_size": 20, "raw_value_size": 14174127, "raw_average_value_size": 4499, "num_data_blocks": 1860, "num_entries": 3150, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:39:27.976099 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 698] Compacted 1@0 + 1@1 files to L1 => 14375354 bytes
>2018-07-11 04:39:28.009624 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:39:28.009555) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 72.8 rd, 63.9 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.1) out(13.7), read-write-amplify(19.6) write-amplify(9.1) OK, records in: 3660, records dropped: 510
>
>2018-07-11 04:39:28.009634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:39:28.009577) EVENT_LOG_v1 {"time_micros": 1531283968009567, "job": 698, "event": "compaction_finished", "compaction_time_micros": 224983, "output_level": 1, "num_output_files": 1, "total_output_size": 14375354, "num_input_records": 3660, "num_output_records": 3150, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:39:28.010025 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283968010008, "job": 698, "event": "table_file_deletion", "file_number": 10173}
>2018-07-11 04:39:28.012837 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283968012822, "job": 698, "event": "table_file_deletion", "file_number": 10171}
>2018-07-11 04:39:28.507605 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215307 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:28.702042 7f29149e4700  1 mon.magna102@0(leader).log v18658 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18658)
>2018-07-11 04:39:33.507899 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215310 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:33.627669 7f29149e4700  1 mon.magna102@0(leader).log v18659 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18659)
>2018-07-11 04:39:41.475420 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215313 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:41.629949 7f29149e4700  1 mon.magna102@0(leader).log v18660 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18660)
>2018-07-11 04:39:47.478089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215316 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:47.630598 7f29149e4700  1 mon.magna102@0(leader).log v18661 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18661)
>2018-07-11 04:39:53.480563 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215319 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:53.631380 7f29149e4700  1 mon.magna102@0(leader).log v18662 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18662)
>2018-07-11 04:39:59.483170 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215322 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:39:59.640286 7f29149e4700  1 mon.magna102@0(leader).log v18663 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18663)
>2018-07-11 04:40:00.241679 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:40:09.487651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215325 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:09.641428 7f29149e4700  1 mon.magna102@0(leader).log v18664 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18664)
>2018-07-11 04:40:18.533410 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215328 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:18.642280 7f29149e4700  1 mon.magna102@0(leader).log v18665 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18665)
>2018-07-11 04:40:23.534236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215331 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:23.651275 7f29149e4700  1 mon.magna102@0(leader).log v18666 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18666)
>2018-07-11 04:40:28.534452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215334 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:28.651811 7f29149e4700  1 mon.magna102@0(leader).log v18667 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18667)
>2018-07-11 04:40:38.544006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215337 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:38.637404 7f29149e4700  1 mon.magna102@0(leader).log v18668 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18668)
>2018-07-11 04:40:43.544237 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215340 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:43.663041 7f29149e4700  1 mon.magna102@0(leader).log v18669 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18669)
>2018-07-11 04:40:48.544451 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215343 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:48.638519 7f29149e4700  1 mon.magna102@0(leader).log v18670 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18670)
>2018-07-11 04:40:53.544686 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215346 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:53.664174 7f29149e4700  1 mon.magna102@0(leader).log v18671 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18671)
>2018-07-11 04:40:59.508979 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215349 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:40:59.656444 7f29149e4700  1 mon.magna102@0(leader).log v18672 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18672)
>2018-07-11 04:41:00.241859 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:41:07.512398 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215355 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:07.665618 7f29149e4700  1 mon.magna102@0(leader).log v18673 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18673)
>2018-07-11 04:41:13.514931 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215358 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:13.666204 7f29149e4700  1 mon.magna102@0(leader).log v18674 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18674)
>2018-07-11 04:41:21.518373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215361 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:21.666812 7f29149e4700  1 mon.magna102@0(leader).log v18675 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18675)
>2018-07-11 04:41:22.733720 7f29149e4700  1 mon.magna102@0(leader).log v18676 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18676)
>2018-07-11 04:41:23.867291 7f29149e4700  1 mon.magna102@0(leader).log v18677 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18677)
>2018-07-11 04:41:27.520995 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215364 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:27.676261 7f29149e4700  1 mon.magna102@0(leader).log v18678 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18678)
>2018-07-11 04:41:32.661209 7f29149e4700  1 mon.magna102@0(leader).log v18679 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18679)
>2018-07-11 04:41:35.524290 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215367 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:35.678416 7f29149e4700  1 mon.magna102@0(leader).log v18680 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18680)
>2018-07-11 04:41:43.578818 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215370 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:43.712587 7f29149e4700  1 mon.magna102@0(leader).log v18681 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18681)
>2018-07-11 04:41:48.579048 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215373 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:48.696376 7f29149e4700  1 mon.magna102@0(leader).log v18682 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18682)
>2018-07-11 04:41:58.579433 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215376 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:41:58.697260 7f29149e4700  1 mon.magna102@0(leader).log v18683 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18683)
>2018-07-11 04:42:00.242046 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:42:03.579663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215379 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:03.706353 7f29149e4700  1 mon.magna102@0(leader).log v18684 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18684)
>2018-07-11 04:42:08.579899 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215382 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:08.681787 7f29149e4700  1 mon.magna102@0(leader).log v18685 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18685)
>2018-07-11 04:42:15.541323 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215388 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:15.699236 7f29149e4700  1 mon.magna102@0(leader).log v18686 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18686)
>2018-07-11 04:42:21.544054 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215391 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:21.699861 7f29149e4700  1 mon.magna102@0(leader).log v18687 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18687)
>2018-07-11 04:42:31.548300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215394 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:31.701957 7f29149e4700  1 mon.magna102@0(leader).log v18688 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18688)
>2018-07-11 04:42:39.551577 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215397 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:39.710995 7f29149e4700  1 mon.magna102@0(leader).log v18689 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18689)
>2018-07-11 04:42:48.586266 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215400 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:48.686704 7f29149e4700  1 mon.magna102@0(leader).log v18690 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18690)
>2018-07-11 04:42:53.587015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215403 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:53.720821 7f29149e4700  1 mon.magna102@0(leader).log v18691 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18691)
>2018-07-11 04:42:58.587226 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215406 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:42:58.696265 7f29149e4700  1 mon.magna102@0(leader).log v18692 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18692)
>2018-07-11 04:43:00.242233 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:43:03.596576 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215409 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:03.713551 7f29149e4700  1 mon.magna102@0(leader).log v18693 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18693)
>2018-07-11 04:43:09.564331 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215412 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:09.714217 7f29149e4700  1 mon.magna102@0(leader).log v18694 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18694)
>2018-07-11 04:43:18.606122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215415 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:18.706585 7f29149e4700  1 mon.magna102@0(leader).log v18695 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18695)
>2018-07-11 04:43:23.606923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215418 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:23.733493 7f29149e4700  1 mon.magna102@0(leader).log v18696 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18696)
>2018-07-11 04:43:28.607120 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215421 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:28.800610 7f29149e4700  1 mon.magna102@0(leader).log v18697 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18697)
>2018-07-11 04:43:38.617758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215424 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:38.734817 7f29149e4700  1 mon.magna102@0(leader).log v18698 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18698)
>2018-07-11 04:43:43.617994 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215427 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:43.735502 7f29149e4700  1 mon.magna102@0(leader).log v18699 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18699)
>2018-07-11 04:43:48.618223 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215430 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:48.723976 7f29149e4700  1 mon.magna102@0(leader).log v18700 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18700)
>2018-07-11 04:43:53.619710 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215433 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:53.736714 7f29149e4700  1 mon.magna102@0(leader).log v18701 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18701)
>2018-07-11 04:43:58.619915 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215436 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:43:58.720496 7f29149e4700  1 mon.magna102@0(leader).log v18702 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18702)
>2018-07-11 04:44:00.242414 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:44:05.588228 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215439 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:05.754802 7f29149e4700  1 mon.magna102@0(leader).log v18703 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18703)
>2018-07-11 04:44:11.590930 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215442 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:11.738714 7f29149e4700  1 mon.magna102@0(leader).log v18704 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18704)
>2018-07-11 04:44:18.630539 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215445 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:18.732208 7f29149e4700  1 mon.magna102@0(leader).log v18705 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18705)
>2018-07-11 04:44:23.632537 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215448 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:23.764096 7f29149e4700  1 mon.magna102@0(leader).log v18706 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18706)
>2018-07-11 04:44:28.632742 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215451 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:28.733304 7f29149e4700  1 mon.magna102@0(leader).log v18707 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18707)
>2018-07-11 04:44:38.633771 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215454 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:38.734215 7f29149e4700  1 mon.magna102@0(leader).log v18708 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18708)
>2018-07-11 04:44:48.643059 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215457 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:48.743488 7f29149e4700  1 mon.magna102@0(leader).log v18709 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18709)
>2018-07-11 04:44:53.643612 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215463 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:53.769096 7f29149e4700  1 mon.magna102@0(leader).log v18710 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18710)
>2018-07-11 04:44:59.611298 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215466 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:44:59.778030 7f29149e4700  1 mon.magna102@0(leader).log v18711 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18711)
>2018-07-11 04:45:00.242595 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:45:07.614899 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215469 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:07.787076 7f29149e4700  1 mon.magna102@0(leader).log v18712 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18712)
>2018-07-11 04:45:08.737770 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10175. Immutable memtables: 0.
>
>2018-07-11 04:45:08.737825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 699] Syncing log #10172
>2018-07-11 04:45:08.754024 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.737815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:45:08.754035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 699] Flushing memtable with next log file: 10175
>
>2018-07-11 04:45:08.754062 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284308754053, "job": 699, "event": "flush_started", "num_memtables": 1, "num_entries": 1885, "num_deletes": 250, "memory_usage": 3590544}
>2018-07-11 04:45:08.754065 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 699] Level-0 flush table #10176: started
>2018-07-11 04:45:08.812914 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284308812900, "cf_name": "default", "job": 699, "event": "table_file_creation", "file_number": 10176, "file_size": 2345097, "table_properties": {"data_size": 2326836, "index_size": 9430, "filter_size": 7907, "raw_key_size": 12616, "raw_average_key_size": 21, "raw_value_size": 2311752, "raw_average_value_size": 3852, "num_data_blocks": 290, "num_entries": 600, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:45:08.812925 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 699] Level-0 flush table #10176: 2345097 bytes OK
>2018-07-11 04:45:08.862920 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.829302) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10176 started
>2018-07-11 04:45:08.862928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.862883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10176: memtable #1 done
>2018-07-11 04:45:08.862930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.862898) EVENT_LOG_v1 {"time_micros": 1531284308862891, "job": 699, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:45:08.862946 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.862910) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:45:08.862952 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 699] Try to delete WAL files size 3553850, prev total WAL file size 3553850, number of live WAL files 2.
>
>2018-07-11 04:45:08.863877 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:45:08.864069 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 04:45:08.864077 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 46335.7 total, 725.2 interval
>Cumulative writes: 65K writes, 312K keys, 65K commit groups, 1.0 writes per commit group, ingest: 0.52 GB, 0.01 MB/s
>Cumulative WAL: 65K writes, 65K syncs, 1.00 writes per sync, written: 0.52 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 954 writes, 4196 keys, 954 commit groups, 1.0 writes per commit group, ingest: 7.12 MB, 0.01 MB/s
>Interval WAL: 954 writes, 954 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.24 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        29       350    0.083       0      0
>  L1      1/1   13.71 MB   0.0      5.5     0.5      5.0       5.0     -0.0       0.0  10.9     72.3     66.2        78       348    0.223   1230K   144K
> Sum      2/2   15.95 MB   0.0      5.5     0.5      5.0       5.5      0.5       0.0  11.9     52.5     52.5       107       698    0.153   1230K   144K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.7     51.3     51.6         1         8    0.147     14K   1427
>Uptime(secs): 46335.7 total, 46335.7 interval
>Flush(GB): cumulative 0.461, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.47 GB write, 0.12 MB/s write, 5.47 GB read, 0.12 MB/s read, 106.6 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.24 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        29       350    0.083       0      0
>  L1      1/1   13.71 MB   0.0      5.5     0.5      5.0       5.0     -0.0       0.0  10.9     72.3     66.2        78       348    0.223   1230K   144K
> Sum      2/2   15.95 MB   0.0      5.5     0.5      5.0       5.5      0.5       0.0  11.9     52.5     52.5       107       698    0.153   1230K   144K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 46335.7 total, 46335.7 interval
>Flush(GB): cumulative 0.461, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.47 GB write, 0.12 MB/s write, 5.47 GB read, 0.12 MB/s read, 106.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 04:45:08.864101 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:08.864095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:45:08.864106 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 700] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:45:08.864111 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 699 Base level 0, inputs: [10176(2290KB)], [10174(13MB)]
>
>2018-07-11 04:45:08.864124 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284308864114, "job": 700, "event": "compaction_started", "files_L0": [10176], "files_L1": [10174], "score": -1, "input_data_size": 16720451}
>2018-07-11 04:45:09.056529 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 700] Generated table #10177: 3406 keys, 16128356 bytes
>2018-07-11 04:45:09.056575 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284309056547, "cf_name": "default", "job": 700, "event": "table_file_creation", "file_number": 10177, "file_size": 16128356, "table_properties": {"data_size": 16004735, "index_size": 70475, "filter_size": 52219, "raw_key_size": 68534, "raw_average_key_size": 20, "raw_value_size": 15906816, "raw_average_value_size": 4670, "num_data_blocks": 2064, "num_entries": 3406, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:45:09.089713 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 700] Compacted 1@0 + 1@1 files to L1 => 16128356 bytes
>2018-07-11 04:45:09.131544 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:09.131476) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.1 rd, 71.5 wr, level 1, files in(1, 1) out(1) MB in(2.2, 13.7) out(15.4), read-write-amplify(14.0) write-amplify(6.9) OK, records in: 3750, records dropped: 344
>
>2018-07-11 04:45:09.131553 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:09.131497) EVENT_LOG_v1 {"time_micros": 1531284309131488, "job": 700, "event": "compaction_finished", "compaction_time_micros": 225573, "output_level": 1, "num_output_files": 1, "total_output_size": 16128356, "num_input_records": 3750, "num_output_records": 3406, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:45:09.132081 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284309132077, "job": 700, "event": "table_file_deletion", "file_number": 10176}
>2018-07-11 04:45:09.134908 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284309134905, "job": 700, "event": "table_file_deletion", "file_number": 10174}
>2018-07-11 04:45:09.135025 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:45:13.695755 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215472 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:14.766028 7f29149e4700  1 mon.magna102@0(leader).log v18713 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18713)
>2018-07-11 04:45:18.695996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215475 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:18.816511 7f29149e4700  1 mon.magna102@0(leader).log v18714 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18714)
>2018-07-11 04:45:27.623559 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215478 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:27.775766 7f29149e4700  1 mon.magna102@0(leader).log v18715 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18715)
>2018-07-11 04:45:33.696598 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215481 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:34.759743 7f29149e4700  1 mon.magna102@0(leader).log v18716 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18716)
>2018-07-11 04:45:38.696825 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215484 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:38.785253 7f29149e4700  1 mon.magna102@0(leader).log v18717 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18717)
>2018-07-11 04:45:45.786170 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10178. Immutable memtables: 0.
>
>2018-07-11 04:45:45.786274 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 701] Syncing log #10175
>2018-07-11 04:45:45.802764 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.786265) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:45:45.802776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 701] Flushing memtable with next log file: 10178
>
>2018-07-11 04:45:45.802789 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284345802781, "job": 701, "event": "flush_started", "num_memtables": 1, "num_entries": 429, "num_deletes": 251, "memory_usage": 396704}
>2018-07-11 04:45:45.802791 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 701] Level-0 flush table #10179: started
>2018-07-11 04:45:45.836643 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284345836630, "cf_name": "default", "job": 701, "event": "table_file_creation", "file_number": 10179, "file_size": 378075, "table_properties": {"data_size": 373704, "index_size": 1628, "filter_size": 1824, "raw_key_size": 6350, "raw_average_key_size": 20, "raw_value_size": 368287, "raw_average_value_size": 1176, "num_data_blocks": 50, "num_entries": 313, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:45:45.836653 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 701] Level-0 flush table #10179: 378075 bytes OK
>2018-07-11 04:45:45.895004 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.853214) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10179 started
>2018-07-11 04:45:45.895011 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.894969) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10179: memtable #1 done
>2018-07-11 04:45:45.895014 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.894983) EVENT_LOG_v1 {"time_micros": 1531284345894977, "job": 701, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:45:45.895016 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.894994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:45:45.895021 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 701] Try to delete WAL files size 386196, prev total WAL file size 386196, number of live WAL files 2.
>
>2018-07-11 04:45:45.895186 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:45:45.895238 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:45.895231) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:45:45.895247 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 702] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:45:45.895253 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 701 Base level 0, inputs: [10179(369KB)], [10177(15MB)]
>
>2018-07-11 04:45:45.895265 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284345895256, "job": 702, "event": "compaction_started", "files_L0": [10179], "files_L1": [10177], "score": -1, "input_data_size": 16506431}
>2018-07-11 04:45:46.079046 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 702] Generated table #10180: 3209 keys, 14510123 bytes
>2018-07-11 04:45:46.079095 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284346079067, "cf_name": "default", "job": 702, "event": "table_file_creation", "file_number": 10180, "file_size": 14510123, "table_properties": {"data_size": 14398034, "index_size": 63885, "filter_size": 47277, "raw_key_size": 64637, "raw_average_key_size": 20, "raw_value_size": 14307471, "raw_average_value_size": 4458, "num_data_blocks": 1862, "num_entries": 3209, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:45:46.112235 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 702] Compacted 1@0 + 1@1 files to L1 => 14510123 bytes
>2018-07-11 04:45:46.154091 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:46.154020) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.1 rd, 66.9 wr, level 1, files in(1, 1) out(1) MB in(0.4, 15.4) out(13.8), read-write-amplify(82.0) write-amplify(38.4) OK, records in: 3719, records dropped: 510
>
>2018-07-11 04:45:46.154100 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:45:46.154041) EVENT_LOG_v1 {"time_micros": 1531284346154032, "job": 702, "event": "compaction_finished", "compaction_time_micros": 216955, "output_level": 1, "num_output_files": 1, "total_output_size": 14510123, "num_input_records": 3719, "num_output_records": 3209, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:45:46.154247 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284346154243, "job": 702, "event": "table_file_deletion", "file_number": 10179}
>2018-07-11 04:45:46.157189 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284346157186, "job": 702, "event": "table_file_deletion", "file_number": 10177}
>2018-07-11 04:45:48.697236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215487 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:48.839426 7f29149e4700  1 mon.magna102@0(leader).log v18718 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18718)
>2018-07-11 04:45:53.697512 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215490 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:53.815101 7f29149e4700  1 mon.magna102@0(leader).log v18719 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18719)
>2018-07-11 04:45:58.697740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215493 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:45:58.798878 7f29149e4700  1 mon.magna102@0(leader).log v18720 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18720)
>2018-07-11 04:46:00.242782 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:46:08.698118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215496 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:08.808209 7f29149e4700  1 mon.magna102@0(leader).log v18721 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18721)
>2018-07-11 04:46:13.698357 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215499 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:13.817305 7f29149e4700  1 mon.magna102@0(leader).log v18722 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18722)
>2018-07-11 04:46:18.698585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215502 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:18.776071 7f29149e4700  1 mon.magna102@0(leader).log v18723 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18723)
>2018-07-11 04:46:25.648300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215505 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:25.801852 7f29149e4700  1 mon.magna102@0(leader).log v18724 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18724)
>2018-07-11 04:46:33.699168 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215508 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:33.819281 7f29149e4700  1 mon.magna102@0(leader).log v18725 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18725)
>2018-07-11 04:46:38.699402 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215511 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:38.786484 7f29149e4700  1 mon.magna102@0(leader).log v18726 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18726)
>2018-07-11 04:46:43.699595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215514 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:43.821810 7f29149e4700  1 mon.magna102@0(leader).log v18727 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18727)
>2018-07-11 04:46:51.659410 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215517 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:51.814218 7f29149e4700  1 mon.magna102@0(leader).log v18728 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18728)
>2018-07-11 04:46:57.661979 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215520 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:46:57.814807 7f29149e4700  1 mon.magna102@0(leader).log v18729 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18729)
>2018-07-11 04:47:00.242970 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:47:03.700326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215523 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:03.823761 7f29149e4700  1 mon.magna102@0(leader).log v18730 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18730)
>2018-07-11 04:47:08.700545 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215526 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:08.799213 7f29149e4700  1 mon.magna102@0(leader).log v18731 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18731)
>2018-07-11 04:47:15.669538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215529 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:15.824980 7f29149e4700  1 mon.magna102@0(leader).log v18732 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18732)
>2018-07-11 04:47:23.673080 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215535 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:23.842375 7f29149e4700  1 mon.magna102@0(leader).log v18733 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18733)
>2018-07-11 04:47:31.676607 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215538 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:31.851428 7f29149e4700  1 mon.magna102@0(leader).log v18734 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18734)
>2018-07-11 04:47:37.679267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215541 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:37.852022 7f29149e4700  1 mon.magna102@0(leader).log v18735 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18735)
>2018-07-11 04:47:43.681608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215544 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:43.828856 7f29149e4700  1 mon.magna102@0(leader).log v18736 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18736)
>2018-07-11 04:47:51.685227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215547 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:51.838014 7f29149e4700  1 mon.magna102@0(leader).log v18737 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18737)
>2018-07-11 04:47:59.688513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215550 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:47:59.838807 7f29149e4700  1 mon.magna102@0(leader).log v18738 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18738)
>2018-07-11 04:48:00.243145 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:48:07.692040 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215556 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:07.839553 7f29149e4700  1 mon.magna102@0(leader).log v18739 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18739)
>2018-07-11 04:48:17.696414 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215559 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:17.848764 7f29149e4700  1 mon.magna102@0(leader).log v18740 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18740)
>2018-07-11 04:48:23.699163 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215562 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:23.849425 7f29149e4700  1 mon.magna102@0(leader).log v18741 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18741)
>2018-07-11 04:48:28.732608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215565 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:28.833102 7f29149e4700  1 mon.magna102@0(leader).log v18742 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18742)
>2018-07-11 04:48:33.741812 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215568 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:33.858860 7f29149e4700  1 mon.magna102@0(leader).log v18743 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18743)
>2018-07-11 04:48:39.705974 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215571 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:39.860659 7f29149e4700  1 mon.magna102@0(leader).log v18744 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18744)
>2018-07-11 04:48:45.708539 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215574 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:45.869610 7f29149e4700  1 mon.magna102@0(leader).log v18745 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18745)
>2018-07-11 04:48:51.711322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215577 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:51.861746 7f29149e4700  1 mon.magna102@0(leader).log v18746 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18746)
>2018-07-11 04:48:58.753704 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215580 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:48:58.879118 7f29149e4700  1 mon.magna102@0(leader).log v18747 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18747)
>2018-07-11 04:49:00.243343 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:49:07.718121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215586 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:07.888381 7f29149e4700  1 mon.magna102@0(leader).log v18748 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18748)
>2018-07-11 04:49:19.723160 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215589 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:19.897771 7f29149e4700  1 mon.magna102@0(leader).log v18749 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18749)
>2018-07-11 04:49:28.764199 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215592 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:28.856824 7f29149e4700  1 mon.magna102@0(leader).log v18750 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18750)
>2018-07-11 04:49:33.765460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215595 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:33.882498 7f29149e4700  1 mon.magna102@0(leader).log v18751 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18751)
>2018-07-11 04:49:38.765674 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215598 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:38.867386 7f29149e4700  1 mon.magna102@0(leader).log v18752 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18752)
>2018-07-11 04:49:43.767902 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215601 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:43.884949 7f29149e4700  1 mon.magna102@0(leader).log v18753 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18753)
>2018-07-11 04:49:48.768099 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215604 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:48.868723 7f29149e4700  1 mon.magna102@0(leader).log v18754 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18754)
>2018-07-11 04:49:58.769195 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215607 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:49:58.877949 7f29149e4700  1 mon.magna102@0(leader).log v18755 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18755)
>2018-07-11 04:50:00.243530 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:50:03.778562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215610 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:03.895604 7f29149e4700  1 mon.magna102@0(leader).log v18756 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18756)
>2018-07-11 04:50:08.778819 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215613 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:08.879386 7f29149e4700  1 mon.magna102@0(leader).log v18757 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18757)
>2018-07-11 04:50:13.779787 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215616 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:13.896695 7f29149e4700  1 mon.magna102@0(leader).log v18758 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18758)
>2018-07-11 04:50:19.749114 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215619 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:19.897500 7f29149e4700  1 mon.magna102@0(leader).log v18759 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18759)
>2018-07-11 04:50:27.752530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215622 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:27.906565 7f29149e4700  1 mon.magna102@0(leader).log v18760 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18760)
>2018-07-11 04:50:33.755116 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215625 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:33.925033 7f29149e4700  1 mon.magna102@0(leader).log v18761 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18761)
>2018-07-11 04:50:38.791565 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215628 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:38.917404 7f29149e4700  1 mon.magna102@0(leader).log v18762 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18762)
>2018-07-11 04:50:48.792851 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215631 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:48.893303 7f29149e4700  1 mon.magna102@0(leader).log v18763 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18763)
>2018-07-11 04:50:53.793602 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215634 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:53.910654 7f29149e4700  1 mon.magna102@0(leader).log v18764 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18764)
>2018-07-11 04:50:58.793796 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215637 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:50:58.902792 7f29149e4700  1 mon.magna102@0(leader).log v18765 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18765)
>2018-07-11 04:51:00.243717 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:51:03.904104 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10181. Immutable memtables: 0.
>
>2018-07-11 04:51:03.904159 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 703] Syncing log #10178
>2018-07-11 04:51:03.920174 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:51:03.904149) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:51:03.920192 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 703] Flushing memtable with next log file: 10181
>
>2018-07-11 04:51:03.920218 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284663920210, "job": 703, "event": "flush_started", "num_memtables": 1, "num_entries": 2002, "num_deletes": 500, "memory_usage": 3324376}
>2018-07-11 04:51:03.920221 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 703] Level-0 flush table #10182: started
>2018-07-11 04:51:03.987383 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284663987368, "cf_name": "default", "job": 703, "event": "table_file_creation", "file_number": 10182, "file_size": 3199837, "table_properties": {"data_size": 3173621, "index_size": 13968, "filter_size": 11323, "raw_key_size": 20082, "raw_average_key_size": 20, "raw_value_size": 3150300, "raw_average_value_size": 3244, "num_data_blocks": 420, "num_entries": 971, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 04:51:03.987393 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 703] Level-0 flush table #10182: 3199837 bytes OK
>2018-07-11 04:51:04.037394 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.003878) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10182 started
>2018-07-11 04:51:04.037402 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.037357) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10182: memtable #1 done
>2018-07-11 04:51:04.037405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.037371) EVENT_LOG_v1 {"time_micros": 1531284664037365, "job": 703, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:51:04.037407 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.037384) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:51:04.037426 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 703] Try to delete WAL files size 3282507, prev total WAL file size 3282507, number of live WAL files 2.
>
>2018-07-11 04:51:04.038247 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:51:04.038297 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.038290) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 04:51:04.038307 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 704] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:51:04.038313 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 703 Base level 0, inputs: [10182(3124KB)], [10180(13MB)]
>
>2018-07-11 04:51:04.038326 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284664038316, "job": 704, "event": "compaction_started", "files_L0": [10182], "files_L1": [10180], "score": -1, "input_data_size": 17709960}
>2018-07-11 04:51:04.221424 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 704] Generated table #10183: 3172 keys, 15444905 bytes
>2018-07-11 04:51:04.221471 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284664221455, "cf_name": "default", "job": 704, "event": "table_file_creation", "file_number": 10183, "file_size": 15444905, "table_properties": {"data_size": 15325204, "index_size": 68721, "filter_size": 50053, "raw_key_size": 64013, "raw_average_key_size": 20, "raw_value_size": 15231793, "raw_average_value_size": 4801, "num_data_blocks": 2017, "num_entries": 3172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:51:04.254616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 704] Compacted 1@0 + 1@1 files to L1 => 15444905 bytes
>2018-07-11 04:51:04.288121 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.288050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.9 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(3.1, 13.8) out(14.7), read-write-amplify(10.4) write-amplify(4.8) OK, records in: 4180, records dropped: 1008
>
>2018-07-11 04:51:04.288130 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:51:04.288071) EVENT_LOG_v1 {"time_micros": 1531284664288062, "job": 704, "event": "compaction_finished", "compaction_time_micros": 216274, "output_level": 1, "num_output_files": 1, "total_output_size": 15444905, "num_input_records": 4180, "num_output_records": 3172, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:51:04.288823 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284664288819, "job": 704, "event": "table_file_deletion", "file_number": 10182}
>2018-07-11 04:51:04.291665 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284664291662, "job": 704, "event": "table_file_deletion", "file_number": 10180}
>2018-07-11 04:51:04.291781 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:51:07.769948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215640 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:07.922365 7f29149e4700  1 mon.magna102@0(leader).log v18766 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18766)
>2018-07-11 04:51:13.807821 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215643 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:13.924250 7f29149e4700  1 mon.magna102@0(leader).log v18767 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18767)
>2018-07-11 04:51:18.808013 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215646 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:18.908024 7f29149e4700  1 mon.magna102@0(leader).log v18768 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18768)
>2018-07-11 04:51:28.816916 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215649 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:28.917373 7f29149e4700  1 mon.magna102@0(leader).log v18769 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18769)
>2018-07-11 04:51:33.817655 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215652 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:33.943014 7f29149e4700  1 mon.magna102@0(leader).log v18770 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18770)
>2018-07-11 04:51:38.817865 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215655 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:38.943435 7f29149e4700  1 mon.magna102@0(leader).log v18771 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18771)
>2018-07-11 04:51:42.952223 7f29149e4700  1 mon.magna102@0(leader).log v18772 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18772)
>2018-07-11 04:51:43.952260 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215658 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:44.127616 7f29149e4700  1 mon.magna102@0(leader).log v18773 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18773)
>2018-07-11 04:51:53.828146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215661 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:53.970230 7f29149e4700  1 mon.magna102@0(leader).log v18774 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18774)
>2018-07-11 04:51:58.828389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215664 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:51:58.937307 7f29149e4700  1 mon.magna102@0(leader).log v18775 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18775)
>2018-07-11 04:52:00.243901 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 04:52:03.829354 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215667 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:03.946398 7f29149e4700  1 mon.magna102@0(leader).log v18776 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18776)
>2018-07-11 04:52:05.939900 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10184. Immutable memtables: 0.
>
>2018-07-11 04:52:05.939942 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 705] Syncing log #10181
>2018-07-11 04:52:05.956478 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:52:05.939933) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:52:05.956489 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 705] Flushing memtable with next log file: 10184
>
>2018-07-11 04:52:05.956516 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284725956508, "job": 705, "event": "flush_started", "num_memtables": 1, "num_entries": 561, "num_deletes": 251, "memory_usage": 691208}
>2018-07-11 04:52:05.956520 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 705] Level-0 flush table #10185: started
>2018-07-11 04:52:06.006998 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726006984, "cf_name": "default", "job": 705, "event": "table_file_creation", "file_number": 10185, "file_size": 663631, "table_properties": {"data_size": 657177, "index_size": 2820, "filter_size": 2715, "raw_key_size": 7220, "raw_average_key_size": 20, "raw_value_size": 650313, "raw_average_value_size": 1831, "num_data_blocks": 86, "num_entries": 355, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:52:06.007008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 705] Level-0 flush table #10185: 663631 bytes OK
>2018-07-11 04:52:06.056970 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.023532) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10185 started
>2018-07-11 04:52:06.056978 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.056935) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10185: memtable #1 done
>2018-07-11 04:52:06.056981 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.056949) EVENT_LOG_v1 {"time_micros": 1531284726056943, "job": 705, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:52:06.056995 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.056960) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:52:06.057002 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 705] Try to delete WAL files size 678193, prev total WAL file size 678193, number of live WAL files 2.
>
>2018-07-11 04:52:06.057233 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:52:06.057282 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.057275) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:52:06.057291 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 706] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:52:06.057297 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 705 Base level 0, inputs: [10185(648KB)], [10183(14MB)]
>
>2018-07-11 04:52:06.057311 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726057301, "job": 706, "event": "compaction_started", "files_L0": [10185], "files_L1": [10183], "score": -1, "input_data_size": 16108536}
>2018-07-11 04:52:06.232716 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 706] Generated table #10186: 3017 keys, 14120876 bytes
>2018-07-11 04:52:06.232760 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726232733, "cf_name": "default", "job": 706, "event": "table_file_creation", "file_number": 10186, "file_size": 14120876, "table_properties": {"data_size": 14010543, "index_size": 63387, "filter_size": 46019, "raw_key_size": 60986, "raw_average_key_size": 20, "raw_value_size": 13923014, "raw_average_value_size": 4614, "num_data_blocks": 1852, "num_entries": 3017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:52:06.274224 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 706] Compacted 1@0 + 1@1 files to L1 => 14120876 bytes
>2018-07-11 04:52:06.307766 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.307695) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.3 rd, 65.1 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.7) out(13.5), read-write-amplify(45.6) write-amplify(21.3) OK, records in: 3527, records dropped: 510
>
>2018-07-11 04:52:06.307775 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:52:06.307717) EVENT_LOG_v1 {"time_micros": 1531284726307707, "job": 706, "event": "compaction_finished", "compaction_time_micros": 216897, "output_level": 1, "num_output_files": 1, "total_output_size": 14120876, "num_input_records": 3527, "num_output_records": 3017, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:52:06.307991 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726307987, "job": 706, "event": "table_file_deletion", "file_number": 10185}
>2018-07-11 04:52:06.310838 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726310835, "job": 706, "event": "table_file_deletion", "file_number": 10183}
>2018-07-11 04:52:08.829556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215670 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:08.933497 7f29149e4700  1 mon.magna102@0(leader).log v18777 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18777)
>2018-07-11 04:52:13.833769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215673 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:13.950803 7f29149e4700  1 mon.magna102@0(leader).log v18778 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18778)
>2018-07-11 04:52:19.176164 7f29149e4700  1 mon.magna102@0(leader).log v18779 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18779)
>2018-07-11 04:52:21.802150 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215676 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:21.951701 7f29149e4700  1 mon.magna102@0(leader).log v18780 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18780)
>2018-07-11 04:52:27.804648 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215679 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:27.952288 7f29149e4700  1 mon.magna102@0(leader).log v18781 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18781)
>2018-07-11 04:52:33.807232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215682 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:33.961159 7f29149e4700  1 mon.magna102@0(leader).log v18782 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18782)
>2018-07-11 04:52:39.809793 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215685 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:39.961904 7f29149e4700  1 mon.magna102@0(leader).log v18783 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18783)
>2018-07-11 04:52:45.812260 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215688 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:45.962460 7f29149e4700  1 mon.magna102@0(leader).log v18784 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18784)
>2018-07-11 04:52:51.815008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215691 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:51.964294 7f29149e4700  1 mon.magna102@0(leader).log v18785 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18785)
>2018-07-11 04:52:57.817587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215694 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:52:57.973191 7f29149e4700  1 mon.magna102@0(leader).log v18786 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18786)
>2018-07-11 04:53:00.244080 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:53:03.820147 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215697 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:03.973756 7f29149e4700  1 mon.magna102@0(leader).log v18787 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18787)
>2018-07-11 04:53:11.823512 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215700 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:11.974441 7f29149e4700  1 mon.magna102@0(leader).log v18788 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18788)
>2018-07-11 04:53:18.857936 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215703 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:18.966682 7f29149e4700  1 mon.magna102@0(leader).log v18789 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18789)
>2018-07-11 04:53:23.866838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215706 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:23.983982 7f29149e4700  1 mon.magna102@0(leader).log v18790 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18790)
>2018-07-11 04:53:28.867029 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215709 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:28.942951 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10187. Immutable memtables: 0.
>
>2018-07-11 04:53:28.942994 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 707] Syncing log #10184
>2018-07-11 04:53:28.959506 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:28.942985) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:53:28.959518 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 707] Flushing memtable with next log file: 10187
>
>2018-07-11 04:53:28.959544 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284808959536, "job": 707, "event": "flush_started", "num_memtables": 1, "num_entries": 654, "num_deletes": 250, "memory_usage": 893712}
>2018-07-11 04:53:28.959547 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 707] Level-0 flush table #10188: started
>2018-07-11 04:53:29.035220 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809035206, "cf_name": "default", "job": 707, "event": "table_file_creation", "file_number": 10188, "file_size": 620223, "table_properties": {"data_size": 614259, "index_size": 2490, "filter_size": 2555, "raw_key_size": 7535, "raw_average_key_size": 21, "raw_value_size": 607713, "raw_average_value_size": 1746, "num_data_blocks": 77, "num_entries": 348, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 04:53:29.035230 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 707] Level-0 flush table #10188: 620223 bytes OK
>2018-07-11 04:53:29.068621 7f29149e4700  1 mon.magna102@0(leader).log v18791 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18791)
>2018-07-11 04:53:29.101884 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.068386) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10188 started
>2018-07-11 04:53:29.101892 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.101849) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10188: memtable #1 done
>2018-07-11 04:53:29.101895 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.101863) EVENT_LOG_v1 {"time_micros": 1531284809101857, "job": 707, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:53:29.101897 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.101875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:53:29.101903 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 707] Try to delete WAL files size 879510, prev total WAL file size 897780, number of live WAL files 2.
>
>2018-07-11 04:53:29.102167 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:53:29.102228 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.102220) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:53:29.102237 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 708] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:53:29.102243 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 707 Base level 0, inputs: [10188(605KB)], [10186(13MB)]
>
>2018-07-11 04:53:29.102256 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809102246, "job": 708, "event": "compaction_started", "files_L0": [10188], "files_L1": [10186], "score": -1, "input_data_size": 14741099}
>2018-07-11 04:53:29.260774 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 708] Generated table #10189: 2892 keys, 13267465 bytes
>2018-07-11 04:53:29.260820 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809260792, "cf_name": "default", "job": 708, "event": "table_file_creation", "file_number": 10189, "file_size": 13267465, "table_properties": {"data_size": 13164911, "index_size": 58470, "filter_size": 43157, "raw_key_size": 58082, "raw_average_key_size": 20, "raw_value_size": 13082682, "raw_average_value_size": 4523, "num_data_blocks": 1714, "num_entries": 2892, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:53:29.294111 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 708] Compacted 1@0 + 1@1 files to L1 => 13267465 bytes
>2018-07-11 04:53:29.327625 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.327554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.8 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(0.6, 13.5) out(12.7), read-write-amplify(45.2) write-amplify(21.4) OK, records in: 3365, records dropped: 473
>
>2018-07-11 04:53:29.327634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.327575) EVENT_LOG_v1 {"time_micros": 1531284809327566, "job": 708, "event": "compaction_finished", "compaction_time_micros": 191840, "output_level": 1, "num_output_files": 1, "total_output_size": 13267465, "num_input_records": 3365, "num_output_records": 2892, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:53:29.327804 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809327800, "job": 708, "event": "table_file_deletion", "file_number": 10188}
>2018-07-11 04:53:29.330594 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809330591, "job": 708, "event": "table_file_deletion", "file_number": 10186}
>2018-07-11 04:53:29.330864 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10190. Immutable memtables: 0.
>
>2018-07-11 04:53:29.330903 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 709] Syncing log #10187
>2018-07-11 04:53:29.344583 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.330896) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:53:29.344595 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 709] Flushing memtable with next log file: 10190
>
>2018-07-11 04:53:29.344608 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809344601, "job": 709, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20632}
>2018-07-11 04:53:29.344610 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 709] Level-0 flush table #10191: started
>2018-07-11 04:53:29.386714 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809386701, "cf_name": "default", "job": 709, "event": "table_file_creation", "file_number": 10191, "file_size": 19385, "table_properties": {"data_size": 18319, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18096, "raw_average_value_size": 2262, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:53:29.386724 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 709] Level-0 flush table #10191: 19385 bytes OK
>2018-07-11 04:53:29.436812 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.403296) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10191 started
>2018-07-11 04:53:29.436820 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.436778) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10191: memtable #1 done
>2018-07-11 04:53:29.436823 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.436792) EVENT_LOG_v1 {"time_micros": 1531284809436786, "job": 709, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:53:29.436825 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.436803) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:53:29.436830 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 709] Try to delete WAL files size 18270, prev total WAL file size 18270, number of live WAL files 2.
>
>2018-07-11 04:53:29.436920 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:53:29.436968 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.436962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 04:53:29.436976 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 710] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:53:29.436982 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 709 Base level 0, inputs: [10191(18KB)], [10189(12MB)]
>
>2018-07-11 04:53:29.436994 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809436985, "job": 710, "event": "compaction_started", "files_L0": [10191], "files_L1": [10189], "score": -1, "input_data_size": 13286850}
>2018-07-11 04:53:29.597095 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 710] Generated table #10192: 2895 keys, 13285730 bytes
>2018-07-11 04:53:29.597143 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809597116, "cf_name": "default", "job": 710, "event": "table_file_creation", "file_number": 10192, "file_size": 13285730, "table_properties": {"data_size": 13183057, "index_size": 58537, "filter_size": 43209, "raw_key_size": 58143, "raw_average_key_size": 20, "raw_value_size": 13100738, "raw_average_value_size": 4525, "num_data_blocks": 1716, "num_entries": 2895, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:53:29.630295 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 710] Compacted 1@0 + 1@1 files to L1 => 13285730 bytes
>2018-07-11 04:53:29.663905 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.663833) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.7 rd, 68.7 wr, level 1, files in(1, 1) out(1) MB in(0.0, 12.7) out(12.7), read-write-amplify(1370.8) write-amplify(685.4) OK, records in: 2900, records dropped: 5
>
>2018-07-11 04:53:29.663916 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:53:29.663857) EVENT_LOG_v1 {"time_micros": 1531284809663847, "job": 710, "event": "compaction_finished", "compaction_time_micros": 193285, "output_level": 1, "num_output_files": 1, "total_output_size": 13285730, "num_input_records": 2900, "num_output_records": 2895, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:53:29.664002 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809663998, "job": 710, "event": "table_file_deletion", "file_number": 10191}
>2018-07-11 04:53:29.666494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284809666491, "job": 710, "event": "table_file_deletion", "file_number": 10189}
>2018-07-11 04:53:33.909599 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215712 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:34.039928 7f29149e4700  1 mon.magna102@0(leader).log v18792 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18792)
>2018-07-11 04:53:41.836636 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215715 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:41.982315 7f29149e4700  1 mon.magna102@0(leader).log v18793 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18793)
>2018-07-11 04:53:45.507514 7f29149e4700  1 mon.magna102@0(leader).log v18794 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18794)
>2018-07-11 04:53:48.910166 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215718 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:49.024720 7f29149e4700  1 mon.magna102@0(leader).log v18795 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18795)
>2018-07-11 04:53:53.910432 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215721 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:54.975420 7f29149e4700  1 mon.magna102@0(leader).log v18796 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18796)
>2018-07-11 04:53:58.910667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215724 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:53:59.025876 7f29149e4700  1 mon.magna102@0(leader).log v18797 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18797)
>2018-07-11 04:54:00.244266 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:54:05.847027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215727 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:06.001631 7f29149e4700  1 mon.magna102@0(leader).log v18798 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18798)
>2018-07-11 04:54:13.911215 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215730 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:14.027368 7f29149e4700  1 mon.magna102@0(leader).log v18799 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18799)
>2018-07-11 04:54:18.911409 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215733 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:18.986105 7f29149e4700  1 mon.magna102@0(leader).log v18800 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18800)
>2018-07-11 04:54:28.911817 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215736 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:28.996651 7f29149e4700  1 mon.magna102@0(leader).log v18801 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18801)
>2018-07-11 04:54:33.912042 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215739 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:34.030715 7f29149e4700  1 mon.magna102@0(leader).log v18802 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18802)
>2018-07-11 04:54:38.912265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215742 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:39.014494 7f29149e4700  1 mon.magna102@0(leader).log v18803 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18803)
>2018-07-11 04:54:43.912500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215745 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:44.040111 7f29149e4700  1 mon.magna102@0(leader).log v18804 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18804)
>2018-07-11 04:54:48.912697 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215748 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:49.032240 7f29149e4700  1 mon.magna102@0(leader).log v18805 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18805)
>2018-07-11 04:54:57.869494 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215751 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:54:58.041431 7f29149e4700  1 mon.magna102@0(leader).log v18806 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18806)
>2018-07-11 04:55:00.244449 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:55:03.913271 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215754 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:04.025466 7f29149e4700  1 mon.magna102@0(leader).log v18807 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18807)
>2018-07-11 04:55:08.913486 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215757 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:09.009233 7f29149e4700  1 mon.magna102@0(leader).log v18808 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18808)
>2018-07-11 04:55:17.878008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215760 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:18.027994 7f29149e4700  1 mon.magna102@0(leader).log v18809 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18809)
>2018-07-11 04:55:23.914076 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215763 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:24.028604 7f29149e4700  1 mon.magna102@0(leader).log v18810 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18810)
>2018-07-11 04:55:29.883023 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215766 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:30.029190 7f29149e4700  1 mon.magna102@0(leader).log v18811 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18811)
>2018-07-11 04:55:38.921122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215769 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:39.021532 7f29149e4700  1 mon.magna102@0(leader).log v18812 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18812)
>2018-07-11 04:55:43.921808 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215772 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:44.038844 7f29149e4700  1 mon.magna102@0(leader).log v18813 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18813)
>2018-07-11 04:55:48.922008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215775 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:49.022606 7f29149e4700  1 mon.magna102@0(leader).log v18814 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18814)
>2018-07-11 04:55:58.931379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215778 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:55:59.031808 7f29149e4700  1 mon.magna102@0(leader).log v18815 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18815)
>2018-07-11 04:56:00.244627 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:56:03.957078 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215781 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:04.074106 7f29149e4700  1 mon.magna102@0(leader).log v18816 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18816)
>2018-07-11 04:56:08.957285 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215784 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:09.032866 7f29149e4700  1 mon.magna102@0(leader).log v18817 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18817)
>2018-07-11 04:56:13.957525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215787 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:14.075292 7f29149e4700  1 mon.magna102@0(leader).log v18818 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18818)
>2018-07-11 04:56:18.957711 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215790 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:19.068601 7f29149e4700  1 mon.magna102@0(leader).log v18819 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18819)
>2018-07-11 04:56:25.907025 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215793 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:26.077669 7f29149e4700  1 mon.magna102@0(leader).log v18820 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18820)
>2018-07-11 04:56:33.958261 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215796 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:34.078415 7f29149e4700  1 mon.magna102@0(leader).log v18821 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18821)
>2018-07-11 04:56:38.958452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215799 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:39.045599 7f29149e4700  1 mon.magna102@0(leader).log v18822 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18822)
>2018-07-11 04:56:48.958848 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215802 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:49.046474 7f29149e4700  1 mon.magna102@0(leader).log v18823 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18823)
>2018-07-11 04:56:53.959088 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215805 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:54.072122 7f29149e4700  1 mon.magna102@0(leader).log v18824 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18824)
>2018-07-11 04:56:58.959297 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215808 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:56:59.055791 7f29149e4700  1 mon.magna102@0(leader).log v18825 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18825)
>2018-07-11 04:57:00.244791 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:57:08.959698 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215811 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:09.056632 7f29149e4700  1 mon.magna102@0(leader).log v18826 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18826)
>2018-07-11 04:57:13.965348 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215814 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:14.083612 7f29149e4700  1 mon.magna102@0(leader).log v18827 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18827)
>2018-07-11 04:57:18.965551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215817 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:19.067254 7f29149e4700  1 mon.magna102@0(leader).log v18828 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18828)
>2018-07-11 04:57:23.967751 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215820 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:24.084784 7f29149e4700  1 mon.magna102@0(leader).log v18829 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18829)
>2018-07-11 04:57:28.967942 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215823 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:29.068645 7f29149e4700  1 mon.magna102@0(leader).log v18830 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18830)
>2018-07-11 04:57:33.977248 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215826 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:34.094291 7f29149e4700  1 mon.magna102@0(leader).log v18831 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18831)
>2018-07-11 04:57:39.938809 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215829 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:40.103190 7f29149e4700  1 mon.magna102@0(leader).log v18832 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18832)
>2018-07-11 04:57:47.942492 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215835 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:48.103899 7f29149e4700  1 mon.magna102@0(leader).log v18833 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18833)
>2018-07-11 04:57:55.945648 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215838 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:57:56.112922 7f29149e4700  1 mon.magna102@0(leader).log v18834 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18834)
>2018-07-11 04:58:00.244970 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 04:58:03.949230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215841 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:04.121952 7f29149e4700  1 mon.magna102@0(leader).log v18835 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18835)
>2018-07-11 04:58:08.988459 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215844 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:09.114040 7f29149e4700  1 mon.magna102@0(leader).log v18836 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18836)
>2018-07-11 04:58:15.954201 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215847 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:16.099358 7f29149e4700  1 mon.magna102@0(leader).log v18837 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18837)
>2018-07-11 04:58:22.092212 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10193. Immutable memtables: 0.
>
>2018-07-11 04:58:22.092257 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 711] Syncing log #10190
>2018-07-11 04:58:22.108748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.092247) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 04:58:22.108759 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 711] Flushing memtable with next log file: 10193
>
>2018-07-11 04:58:22.108786 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102108777, "job": 711, "event": "flush_started", "num_memtables": 1, "num_entries": 1653, "num_deletes": 251, "memory_usage": 3070952}
>2018-07-11 04:58:22.108789 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 711] Level-0 flush table #10194: started
>2018-07-11 04:58:22.167626 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102167612, "cf_name": "default", "job": 711, "event": "table_file_creation", "file_number": 10194, "file_size": 2961012, "table_properties": {"data_size": 2937158, "index_size": 12951, "filter_size": 9979, "raw_key_size": 14227, "raw_average_key_size": 20, "raw_value_size": 2918251, "raw_average_value_size": 4217, "num_data_blocks": 390, "num_entries": 692, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 04:58:22.167638 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 711] Level-0 flush table #10194: 2961012 bytes OK
>2018-07-11 04:58:22.217616 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.184151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10194 started
>2018-07-11 04:58:22.217624 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.217579) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10194: memtable #1 done
>2018-07-11 04:58:22.217627 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.217594) EVENT_LOG_v1 {"time_micros": 1531285102217588, "job": 711, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 04:58:22.217642 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.217606) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 04:58:22.217649 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 711] Try to delete WAL files size 3037605, prev total WAL file size 3037605, number of live WAL files 2.
>
>2018-07-11 04:58:22.218430 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 04:58:22.218641 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 04:58:22.218650 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 47129.1 total, 793.4 interval
>Cumulative writes: 66K writes, 317K keys, 66K commit groups, 1.0 writes per commit group, ingest: 0.53 GB, 0.01 MB/s
>Cumulative WAL: 66K writes, 66K syncs, 1.00 writes per sync, written: 0.53 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1054 writes, 5307 keys, 1054 commit groups, 1.0 writes per commit group, ingest: 7.90 MB, 0.01 MB/s
>Interval WAL: 1054 writes, 1054 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.82 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        30       356    0.083       0      0
>  L1      1/1   12.67 MB   0.0      5.6     0.5      5.1       5.1     -0.0       0.0  10.9     72.3     66.2        79       354    0.222   1251K   147K
> Sum      2/2   15.49 MB   0.0      5.6     0.5      5.1       5.6      0.5       0.0  11.9     52.5     52.5       108       710    0.153   1251K   147K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  12.1     53.2     52.9         2        12    0.142     21K   2850
>Uptime(secs): 47129.1 total, 47129.1 interval
>Flush(GB): cumulative 0.469, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.56 GB write, 0.12 MB/s write, 5.56 GB read, 0.12 MB/s read, 108.3 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.09 GB read, 0.00 MB/s read, 1.7 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.82 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        30       356    0.083       0      0
>  L1      1/1   12.67 MB   0.0      5.6     0.5      5.1       5.1     -0.0       0.0  10.9     72.3     66.2        79       354    0.222   1251K   147K
> Sum      2/2   15.49 MB   0.0      5.6     0.5      5.1       5.6      0.5       0.0  11.9     52.5     52.5       108       710    0.153   1251K   147K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 47129.1 total, 47129.1 interval
>Flush(GB): cumulative 0.469, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.56 GB write, 0.12 MB/s write, 5.56 GB read, 0.12 MB/s read, 108.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 04:58:22.218673 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.218665) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 04:58:22.218678 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 712] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 04:58:22.218683 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 711 Base level 0, inputs: [10194(2891KB)], [10192(12MB)]
>
>2018-07-11 04:58:22.218697 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102218686, "job": 712, "event": "compaction_started", "files_L0": [10194], "files_L1": [10192], "score": -1, "input_data_size": 16246742}
>2018-07-11 04:58:22.393346 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 712] Generated table #10195: 3077 keys, 14247461 bytes
>2018-07-11 04:58:22.393394 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102393366, "cf_name": "default", "job": 712, "event": "table_file_creation", "file_number": 10195, "file_size": 14247461, "table_properties": {"data_size": 14136617, "index_size": 63502, "filter_size": 46415, "raw_key_size": 62123, "raw_average_key_size": 20, "raw_value_size": 14048163, "raw_average_value_size": 4565, "num_data_blocks": 1854, "num_entries": 3077, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 04:58:22.426508 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 712] Compacted 1@0 + 1@1 files to L1 => 14247461 bytes
>2018-07-11 04:58:22.460056 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.459990) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 78.2 rd, 68.6 wr, level 1, files in(1, 1) out(1) MB in(2.8, 12.7) out(13.6), read-write-amplify(10.3) write-amplify(4.8) OK, records in: 3587, records dropped: 510
>
>2018-07-11 04:58:22.460065 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-04:58:22.460011) EVENT_LOG_v1 {"time_micros": 1531285102460002, "job": 712, "event": "compaction_finished", "compaction_time_micros": 207795, "output_level": 1, "num_output_files": 1, "total_output_size": 14247461, "num_input_records": 3587, "num_output_records": 3077, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 04:58:22.460721 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102460717, "job": 712, "event": "table_file_deletion", "file_number": 10194}
>2018-07-11 04:58:22.463371 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285102463368, "job": 712, "event": "table_file_deletion", "file_number": 10192}
>2018-07-11 04:58:23.958226 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215850 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:24.110808 7f29149e4700  1 mon.magna102@0(leader).log v18838 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18838)
>2018-07-11 04:58:28.993816 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215853 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:29.094559 7f29149e4700  1 mon.magna102@0(leader).log v18839 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18839)
>2018-07-11 04:58:33.994948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215856 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:34.111969 7f29149e4700  1 mon.magna102@0(leader).log v18840 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18840)
>2018-07-11 04:58:38.995160 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215859 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:39.095733 7f29149e4700  1 mon.magna102@0(leader).log v18841 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18841)
>2018-07-11 04:58:47.968569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215862 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:48.121602 7f29149e4700  1 mon.magna102@0(leader).log v18842 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18842)
>2018-07-11 04:58:53.971243 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215865 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:58:54.123416 7f29149e4700  1 mon.magna102@0(leader).log v18843 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18843)
>2018-07-11 04:59:00.245160 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 04:59:01.974587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215868 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:02.124120 7f29149e4700  1 mon.magna102@0(leader).log v18844 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18844)
>2018-07-11 04:59:07.977260 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215871 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:08.133096 7f29149e4700  1 mon.magna102@0(leader).log v18845 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18845)
>2018-07-11 04:59:13.979655 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215874 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:14.142005 7f29149e4700  1 mon.magna102@0(leader).log v18846 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18846)
>2018-07-11 04:59:19.016846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215877 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:19.142442 7f29149e4700  1 mon.magna102@0(leader).log v18847 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18847)
>2018-07-11 04:59:27.985717 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215880 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:28.151612 7f29149e4700  1 mon.magna102@0(leader).log v18848 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18848)
>2018-07-11 04:59:34.026890 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215883 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:34.160588 7f29149e4700  1 mon.magna102@0(leader).log v18849 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18849)
>2018-07-11 04:59:39.027097 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215886 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:39.152675 7f29149e4700  1 mon.magna102@0(leader).log v18850 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18850)
>2018-07-11 04:59:44.027964 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215889 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:44.161660 7f29149e4700  1 mon.magna102@0(leader).log v18851 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18851)
>2018-07-11 04:59:51.995966 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215892 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:52.172023 7f29149e4700  1 mon.magna102@0(leader).log v18852 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18852)
>2018-07-11 04:59:59.030507 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215895 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 04:59:59.130925 7f29149e4700  1 mon.magna102@0(leader).log v18853 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18853)
>2018-07-11 05:00:00.245342 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:00:04.031545 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215898 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:04.148548 7f29149e4700  1 mon.magna102@0(leader).log v18854 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18854)
>2018-07-11 05:00:09.031777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215901 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:09.148972 7f29149e4700  1 mon.magna102@0(leader).log v18855 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18855)
>2018-07-11 05:00:19.041095 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215904 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:19.141510 7f29149e4700  1 mon.magna102@0(leader).log v18856 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18856)
>2018-07-11 05:00:24.041856 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215907 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:24.158902 7f29149e4700  1 mon.magna102@0(leader).log v18857 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18857)
>2018-07-11 05:00:29.042066 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215910 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:29.150979 7f29149e4700  1 mon.magna102@0(leader).log v18858 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18858)
>2018-07-11 05:00:38.016067 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215916 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:38.168783 7f29149e4700  1 mon.magna102@0(leader).log v18859 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18859)
>2018-07-11 05:00:44.018556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215919 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:44.169448 7f29149e4700  1 mon.magna102@0(leader).log v18860 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18860)
>2018-07-11 05:00:54.022939 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215922 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:54.196507 7f29149e4700  1 mon.magna102@0(leader).log v18861 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18861)
>2018-07-11 05:00:59.063014 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215925 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:00:59.188528 7f29149e4700  1 mon.magna102@0(leader).log v18862 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18862)
>2018-07-11 05:01:00.245523 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:01:04.064022 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215928 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:04.189376 7f29149e4700  1 mon.magna102@0(leader).log v18863 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18863)
>2018-07-11 05:01:12.030789 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215931 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:12.181711 7f29149e4700  1 mon.magna102@0(leader).log v18864 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18864)
>2018-07-11 05:01:18.033484 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215934 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:18.182287 7f29149e4700  1 mon.magna102@0(leader).log v18865 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18865)
>2018-07-11 05:01:24.036048 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215937 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:24.182849 7f29149e4700  1 mon.magna102@0(leader).log v18866 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18866)
>2018-07-11 05:01:30.038471 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215940 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:30.183478 7f29149e4700  1 mon.magna102@0(leader).log v18867 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18867)
>2018-07-11 05:01:38.042109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215943 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:38.192508 7f29149e4700  1 mon.magna102@0(leader).log v18868 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18868)
>2018-07-11 05:01:44.044543 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215946 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:44.193126 7f29149e4700  1 mon.magna102@0(leader).log v18869 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18869)
>2018-07-11 05:01:49.145005 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10196. Immutable memtables: 0.
>
>2018-07-11 05:01:49.145048 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 713] Syncing log #10193
>2018-07-11 05:01:49.161516 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.145039) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:01:49.161527 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 713] Flushing memtable with next log file: 10196
>
>2018-07-11 05:01:49.161554 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309161546, "job": 713, "event": "flush_started", "num_memtables": 1, "num_entries": 1232, "num_deletes": 250, "memory_usage": 2167448}
>2018-07-11 05:01:49.161557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 713] Level-0 flush table #10197: started
>2018-07-11 05:01:49.212042 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309212028, "cf_name": "default", "job": 713, "event": "table_file_creation", "file_number": 10197, "file_size": 1426587, "table_properties": {"data_size": 1414921, "index_size": 5698, "filter_size": 5048, "raw_key_size": 9873, "raw_average_key_size": 21, "raw_value_size": 1404423, "raw_average_value_size": 3026, "num_data_blocks": 176, "num_entries": 464, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:01:49.212052 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 713] Level-0 flush table #10197: 1426587 bytes OK
>2018-07-11 05:01:49.270391 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.228570) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10197 started
>2018-07-11 05:01:49.270399 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.270355) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10197: memtable #1 done
>2018-07-11 05:01:49.270401 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.270369) EVENT_LOG_v1 {"time_micros": 1531285309270363, "job": 713, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:01:49.270415 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.270381) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:01:49.270422 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 713] Try to delete WAL files size 2142452, prev total WAL file size 2142452, number of live WAL files 2.
>
>2018-07-11 05:01:49.271017 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:01:49.271079 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.271073) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:01:49.271086 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 714] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:01:49.271092 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 713 Base level 0, inputs: [10197(1393KB)], [10195(13MB)]
>
>2018-07-11 05:01:49.271105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309271096, "job": 714, "event": "compaction_started", "files_L0": [10197], "files_L1": [10195], "score": -1, "input_data_size": 15674048}
>2018-07-11 05:01:49.454437 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 714] Generated table #10198: 3130 keys, 14623868 bytes
>2018-07-11 05:01:49.454473 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309454458, "cf_name": "default", "job": 714, "event": "table_file_creation", "file_number": 10198, "file_size": 14623868, "table_properties": {"data_size": 14511460, "index_size": 64057, "filter_size": 47424, "raw_key_size": 62921, "raw_average_key_size": 20, "raw_value_size": 14421937, "raw_average_value_size": 4607, "num_data_blocks": 1877, "num_entries": 3130, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:01:49.495950 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 714] Compacted 1@0 + 1@1 files to L1 => 14623868 bytes
>2018-07-11 05:01:49.537803 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.537734) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.7 rd, 65.0 wr, level 1, files in(1, 1) out(1) MB in(1.4, 13.6) out(13.9), read-write-amplify(21.2) write-amplify(10.3) OK, records in: 3541, records dropped: 411
>
>2018-07-11 05:01:49.537812 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:01:49.537755) EVENT_LOG_v1 {"time_micros": 1531285309537745, "job": 714, "event": "compaction_finished", "compaction_time_micros": 224826, "output_level": 1, "num_output_files": 1, "total_output_size": 14623868, "num_input_records": 3541, "num_output_records": 3130, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:01:49.538191 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309538187, "job": 714, "event": "table_file_deletion", "file_number": 10197}
>2018-07-11 05:01:49.540837 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285309540834, "job": 714, "event": "table_file_deletion", "file_number": 10195}
>2018-07-11 05:01:49.540955 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:01:52.048302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215949 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:52.188655 7f29149e4700  1 mon.magna102@0(leader).log v18870 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18870)
>2018-07-11 05:01:58.050942 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215952 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:01:58.197668 7f29149e4700  1 mon.magna102@0(leader).log v18871 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18871)
>2018-07-11 05:02:00.245704 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:02:06.054254 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215955 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:06.206796 7f29149e4700  1 mon.magna102@0(leader).log v18872 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18872)
>2018-07-11 05:02:14.112279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215958 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:14.232607 7f29149e4700  1 mon.magna102@0(leader).log v18873 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18873)
>2018-07-11 05:02:19.112471 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215964 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:19.191366 7f29149e4700  1 mon.magna102@0(leader).log v18874 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18874)
>2018-07-11 05:02:23.191771 7f29149e4700  1 mon.magna102@0(leader).log v18875 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18875)
>2018-07-11 05:02:24.191823 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215967 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:24.325457 7f29149e4700  1 mon.magna102@0(leader).log v18876 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18876)
>2018-07-11 05:02:25.392239 7f29149e4700  1 mon.magna102@0(leader).log v18877 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18877)
>2018-07-11 05:02:34.066293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215970 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:34.227881 7f29149e4700  1 mon.magna102@0(leader).log v18878 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18878)
>2018-07-11 05:02:39.113196 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215973 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:39.203319 7f29149e4700  1 mon.magna102@0(leader).log v18879 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18879)
>2018-07-11 05:02:49.113571 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215976 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:49.220840 7f29149e4700  1 mon.magna102@0(leader).log v18880 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18880)
>2018-07-11 05:02:58.076617 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215982 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:02:58.280042 7f29149e4700  1 mon.magna102@0(leader).log v18881 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18881)
>2018-07-11 05:03:00.245891 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:03:04.121942 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215985 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:04.255625 7f29149e4700  1 mon.magna102@0(leader).log v18882 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18882)
>2018-07-11 05:03:09.122131 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215988 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:09.247673 7f29149e4700  1 mon.magna102@0(leader).log v18883 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18883)
>2018-07-11 05:03:16.084298 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215991 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:16.256811 7f29149e4700  1 mon.magna102@0(leader).log v18884 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18884)
>2018-07-11 05:03:24.087896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215994 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:24.257612 7f29149e4700  1 mon.magna102@0(leader).log v18885 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18885)
>2018-07-11 05:03:29.124128 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24215997 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:29.250945 7f29149e4700  1 mon.magna102@0(leader).log v18886 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18886)
>2018-07-11 05:03:39.126390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216000 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:39.226807 7f29149e4700  1 mon.magna102@0(leader).log v18887 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18887)
>2018-07-11 05:03:44.127133 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216003 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:44.244072 7f29149e4700  1 mon.magna102@0(leader).log v18888 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18888)
>2018-07-11 05:03:49.127326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216006 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:49.236225 7f29149e4700  1 mon.magna102@0(leader).log v18889 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18889)
>2018-07-11 05:03:54.136383 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216009 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:54.253526 7f29149e4700  1 mon.magna102@0(leader).log v18890 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18890)
>2018-07-11 05:03:59.136614 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216012 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:03:59.237325 7f29149e4700  1 mon.magna102@0(leader).log v18891 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18891)
>2018-07-11 05:04:00.270681 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:04:09.137762 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216015 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:09.238164 7f29149e4700  1 mon.magna102@0(leader).log v18892 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18892)
>2018-07-11 05:04:14.146783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216018 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:14.263829 7f29149e4700  1 mon.magna102@0(leader).log v18893 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18893)
>2018-07-11 05:04:19.147007 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216021 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:19.247592 7f29149e4700  1 mon.magna102@0(leader).log v18894 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18894)
>2018-07-11 05:04:28.115294 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216027 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:28.266339 7f29149e4700  1 mon.magna102@0(leader).log v18895 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18895)
>2018-07-11 05:04:36.118569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216030 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:36.292166 7f29149e4700  1 mon.magna102@0(leader).log v18896 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18896)
>2018-07-11 05:04:42.121258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216033 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:42.284585 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10199. Immutable memtables: 0.
>
>2018-07-11 05:04:42.326285 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.284644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:04:42.326298 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 715] Flushing memtable with next log file: 10199
>
>2018-07-11 05:04:42.326314 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482326305, "job": 715, "event": "flush_started", "num_memtables": 1, "num_entries": 1081, "num_deletes": 251, "memory_usage": 1827688}
>2018-07-11 05:04:42.326317 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 715] Level-0 flush table #10200: started
>2018-07-11 05:04:42.396197 7f29149e4700  1 mon.magna102@0(leader).log v18897 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18897)
>2018-07-11 05:04:42.418462 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482418447, "cf_name": "default", "job": 715, "event": "table_file_creation", "file_number": 10200, "file_size": 1760628, "table_properties": {"data_size": 1745927, "index_size": 7601, "filter_size": 6180, "raw_key_size": 10548, "raw_average_key_size": 20, "raw_value_size": 1733331, "raw_average_value_size": 3365, "num_data_blocks": 231, "num_entries": 515, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:04:42.418473 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 715] Level-0 flush table #10200: 1760628 bytes OK
>2018-07-11 05:04:42.468544 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.435084) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10200 started
>2018-07-11 05:04:42.468553 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.468507) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10200: memtable #1 done
>2018-07-11 05:04:42.468555 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.468522) EVENT_LOG_v1 {"time_micros": 1531285482468516, "job": 715, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:04:42.468557 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.468534) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:04:42.468576 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 715] Try to delete WAL files size 1805178, prev total WAL file size 1823608, number of live WAL files 2.
>
>2018-07-11 05:04:42.469091 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:04:42.469154 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.469146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:04:42.469163 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 716] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:04:42.469169 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 715 Base level 0, inputs: [10200(1719KB)], [10198(13MB)]
>
>2018-07-11 05:04:42.469193 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482469173, "job": 716, "event": "compaction_started", "files_L0": [10200], "files_L1": [10198], "score": -1, "input_data_size": 16384496}
>2018-07-11 05:04:42.644277 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 716] Generated table #10201: 3135 keys, 14372325 bytes
>2018-07-11 05:04:42.644322 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482644306, "cf_name": "default", "job": 716, "event": "table_file_creation", "file_number": 10201, "file_size": 14372325, "table_properties": {"data_size": 14260980, "index_size": 63617, "filter_size": 46801, "raw_key_size": 63222, "raw_average_key_size": 20, "raw_value_size": 14171631, "raw_average_value_size": 4520, "num_data_blocks": 1856, "num_entries": 3135, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:04:42.677413 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 716] Compacted 1@0 + 1@1 files to L1 => 14372325 bytes
>2018-07-11 05:04:42.710951 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.710883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 78.7 rd, 69.0 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.9) out(13.7), read-write-amplify(17.5) write-amplify(8.2) OK, records in: 3645, records dropped: 510
>
>2018-07-11 05:04:42.710960 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:04:42.710904) EVENT_LOG_v1 {"time_micros": 1531285482710895, "job": 716, "event": "compaction_finished", "compaction_time_micros": 208204, "output_level": 1, "num_output_files": 1, "total_output_size": 14372325, "num_input_records": 3645, "num_output_records": 3135, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:04:42.711420 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482711416, "job": 716, "event": "table_file_deletion", "file_number": 10200}
>2018-07-11 05:04:42.714270 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482714267, "job": 716, "event": "table_file_deletion", "file_number": 10198}
>2018-07-11 05:04:48.124024 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216036 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:48.278700 7f29149e4700  1 mon.magna102@0(leader).log v18898 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18898)
>2018-07-11 05:04:54.162282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216039 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:04:54.279314 7f29149e4700  1 mon.magna102@0(leader).log v18899 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18899)
>2018-07-11 05:05:00.129040 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216042 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:00.270860 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:05:00.279825 7f29149e4700  1 mon.magna102@0(leader).log v18900 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18900)
>2018-07-11 05:05:08.132631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216045 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:08.280564 7f29149e4700  1 mon.magna102@0(leader).log v18901 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18901)
>2018-07-11 05:05:14.173894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216048 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:14.290932 7f29149e4700  1 mon.magna102@0(leader).log v18902 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18902)
>2018-07-11 05:05:19.174095 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216051 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:19.274701 7f29149e4700  1 mon.magna102@0(leader).log v18903 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18903)
>2018-07-11 05:05:26.140149 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216054 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:26.292076 7f29149e4700  1 mon.magna102@0(leader).log v18904 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18904)
>2018-07-11 05:05:34.143633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216057 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:34.292806 7f29149e4700  1 mon.magna102@0(leader).log v18905 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18905)
>2018-07-11 05:05:39.175940 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216060 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:39.284902 7f29149e4700  1 mon.magna102@0(leader).log v18906 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18906)
>2018-07-11 05:05:44.185189 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216063 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:44.302215 7f29149e4700  1 mon.magna102@0(leader).log v18907 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18907)
>2018-07-11 05:05:52.151340 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216066 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:52.302835 7f29149e4700  1 mon.magna102@0(leader).log v18908 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18908)
>2018-07-11 05:05:58.153854 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216069 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:05:58.311810 7f29149e4700  1 mon.magna102@0(leader).log v18909 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18909)
>2018-07-11 05:06:00.271047 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:06:04.156439 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216072 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:04.329051 7f29149e4700  1 mon.magna102@0(leader).log v18910 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18910)
>2018-07-11 05:06:12.159957 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216075 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:12.314337 7f29149e4700  1 mon.magna102@0(leader).log v18911 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18911)
>2018-07-11 05:06:18.162432 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216078 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:18.314885 7f29149e4700  1 mon.magna102@0(leader).log v18912 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18912)
>2018-07-11 05:06:24.164935 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216081 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:24.315464 7f29149e4700  1 mon.magna102@0(leader).log v18913 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18913)
>2018-07-11 05:06:29.198613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216084 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:29.299340 7f29149e4700  1 mon.magna102@0(leader).log v18914 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18914)
>2018-07-11 05:06:36.170011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216087 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:36.316827 7f29149e4700  1 mon.magna102@0(leader).log v18915 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18915)
>2018-07-11 05:06:44.173515 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216090 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:44.325859 7f29149e4700  1 mon.magna102@0(leader).log v18916 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18916)
>2018-07-11 05:06:49.209019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216093 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:49.309610 7f29149e4700  1 mon.magna102@0(leader).log v18917 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18917)
>2018-07-11 05:06:58.179591 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216096 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:06:58.327159 7f29149e4700  1 mon.magna102@0(leader).log v18918 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18918)
>2018-07-11 05:07:00.271230 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:07:04.182241 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216102 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:04.336058 7f29149e4700  1 mon.magna102@0(leader).log v18919 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18919)
>2018-07-11 05:07:14.186445 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216105 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:14.338139 7f29149e4700  1 mon.magna102@0(leader).log v18920 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18920)
>2018-07-11 05:07:19.221344 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216108 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:19.330230 7f29149e4700  1 mon.magna102@0(leader).log v18921 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18921)
>2018-07-11 05:07:28.192409 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216111 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:28.364429 7f29149e4700  1 mon.magna102@0(leader).log v18922 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18922)
>2018-07-11 05:07:34.194966 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216114 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:34.365002 7f29149e4700  1 mon.magna102@0(leader).log v18923 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18923)
>2018-07-11 05:07:39.231476 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216117 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:39.348957 7f29149e4700  1 mon.magna102@0(leader).log v18924 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18924)
>2018-07-11 05:07:49.241082 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216120 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:49.366476 7f29149e4700  1 mon.magna102@0(leader).log v18925 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18925)
>2018-07-11 05:07:54.266626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216123 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:54.400453 7f29149e4700  1 mon.magna102@0(leader).log v18926 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18926)
>2018-07-11 05:07:59.266842 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216126 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:07:59.367540 7f29149e4700  1 mon.magna102@0(leader).log v18927 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18927)
>2018-07-11 05:08:00.271411 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:08:08.209404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216132 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:08.368386 7f29149e4700  1 mon.magna102@0(leader).log v18928 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18928)
>2018-07-11 05:08:18.213712 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216135 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:18.362165 7f29149e4700  1 mon.magna102@0(leader).log v18929 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18929)
>2018-07-11 05:08:24.267745 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216138 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:24.388095 7f29149e4700  1 mon.magna102@0(leader).log v18930 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18930)
>2018-07-11 05:08:30.218785 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216141 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:30.371914 7f29149e4700  1 mon.magna102@0(leader).log v18931 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18931)
>2018-07-11 05:08:39.268278 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216144 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:39.356024 7f29149e4700  1 mon.magna102@0(leader).log v18932 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18932)
>2018-07-11 05:08:44.268518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216147 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:44.389960 7f29149e4700  1 mon.magna102@0(leader).log v18933 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18933)
>2018-07-11 05:08:49.268740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216150 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:49.365401 7f29149e4700  1 mon.magna102@0(leader).log v18934 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18934)
>2018-07-11 05:08:54.268978 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216153 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:08:54.391354 7f29149e4700  1 mon.magna102@0(leader).log v18935 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18935)
>2018-07-11 05:09:00.231588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216156 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:00.271591 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:09:00.383631 7f29149e4700  1 mon.magna102@0(leader).log v18936 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18936)
>2018-07-11 05:09:09.269522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216159 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:09.402212 7f29149e4700  1 mon.magna102@0(leader).log v18937 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18937)
>2018-07-11 05:09:16.238540 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216165 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:16.411289 7f29149e4700  1 mon.magna102@0(leader).log v18938 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18938)
>2018-07-11 05:09:22.241198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216168 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:22.386854 7f29149e4700  1 mon.magna102@0(leader).log v18939 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18939)
>2018-07-11 05:09:28.243739 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216171 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:28.395764 7f29149e4700  1 mon.magna102@0(leader).log v18940 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18940)
>2018-07-11 05:09:38.248109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216174 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:38.404913 7f29149e4700  1 mon.magna102@0(leader).log v18941 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18941)
>2018-07-11 05:09:44.250608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216177 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:44.397145 7f29149e4700  1 mon.magna102@0(leader).log v18942 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18942)
>2018-07-11 05:09:49.288221 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216180 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:49.380927 7f29149e4700  1 mon.magna102@0(leader).log v18943 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18943)
>2018-07-11 05:09:56.255759 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216183 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:09:56.406665 7f29149e4700  1 mon.magna102@0(leader).log v18944 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18944)
>2018-07-11 05:10:00.290088 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:10:02.258591 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216186 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:02.407576 7f29149e4700  1 mon.magna102@0(leader).log v18945 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18945)
>2018-07-11 05:10:08.261105 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216189 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:08.409412 7f29149e4700  1 mon.magna102@0(leader).log v18946 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18946)
>2018-07-11 05:10:09.359918 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10202. Immutable memtables: 0.
>
>2018-07-11 05:10:09.359962 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 717] Syncing log #10199
>2018-07-11 05:10:09.376329 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.359952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:10:09.376341 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 717] Flushing memtable with next log file: 10202
>
>2018-07-11 05:10:09.376367 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809376359, "job": 717, "event": "flush_started", "num_memtables": 1, "num_entries": 1796, "num_deletes": 250, "memory_usage": 3399760}
>2018-07-11 05:10:09.376370 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 717] Level-0 flush table #10203: started
>2018-07-11 05:10:09.435207 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809435193, "cf_name": "default", "job": 717, "event": "table_file_creation", "file_number": 10203, "file_size": 2206638, "table_properties": {"data_size": 2189389, "index_size": 8855, "filter_size": 7470, "raw_key_size": 12171, "raw_average_key_size": 21, "raw_value_size": 2175015, "raw_average_value_size": 3763, "num_data_blocks": 273, "num_entries": 578, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:10:09.435217 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 717] Level-0 flush table #10203: 2206638 bytes OK
>2018-07-11 05:10:09.485227 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.451722) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10203 started
>2018-07-11 05:10:09.485236 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.485191) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10203: memtable #1 done
>2018-07-11 05:10:09.485238 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.485206) EVENT_LOG_v1 {"time_micros": 1531285809485199, "job": 717, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:10:09.485253 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.485218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:10:09.485260 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 717] Try to delete WAL files size 3364404, prev total WAL file size 3364404, number of live WAL files 2.
>
>2018-07-11 05:10:09.486100 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:10:09.486302 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 05:10:09.486309 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 47836.3 total, 707.3 interval
>Cumulative writes: 67K writes, 321K keys, 67K commit groups, 1.0 writes per commit group, ingest: 0.54 GB, 0.01 MB/s
>Cumulative WAL: 67K writes, 67K syncs, 1.00 writes per sync, written: 0.54 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 930 writes, 4109 keys, 930 commit groups, 1.0 writes per commit group, ingest: 6.97 MB, 0.01 MB/s
>Interval WAL: 930 writes, 930 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.10 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        30       359    0.083       0      0
>  L1      1/1   13.71 MB   0.0      5.6     0.5      5.1       5.1     -0.0       0.0  10.9     72.3     66.2        79       357    0.222   1262K   149K
> Sum      2/2   15.81 MB   0.0      5.6     0.5      5.1       5.6      0.5       0.0  11.8     52.5     52.5       109       716    0.153   1262K   149K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.0     51.6     52.0         1         6    0.149     10K   1431
>Uptime(secs): 47836.3 total, 47836.3 interval
>Flush(GB): cumulative 0.474, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.60 GB write, 0.12 MB/s write, 5.60 GB read, 0.12 MB/s read, 109.2 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.10 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        30       359    0.083       0      0
>  L1      1/1   13.71 MB   0.0      5.6     0.5      5.1       5.1     -0.0       0.0  10.9     72.3     66.2        79       357    0.222   1262K   149K
> Sum      2/2   15.81 MB   0.0      5.6     0.5      5.1       5.6      0.5       0.0  11.8     52.5     52.5       109       716    0.153   1262K   149K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 47836.3 total, 47836.3 interval
>Flush(GB): cumulative 0.474, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.60 GB write, 0.12 MB/s write, 5.60 GB read, 0.12 MB/s read, 109.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 05:10:09.486330 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.486323) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:10:09.486335 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 718] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:10:09.486340 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 717 Base level 0, inputs: [10203(2154KB)], [10201(13MB)]
>
>2018-07-11 05:10:09.486353 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809486343, "job": 718, "event": "compaction_started", "files_L0": [10203], "files_L1": [10201], "score": -1, "input_data_size": 16578963}
>2018-07-11 05:10:09.677590 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 718] Generated table #10204: 3362 keys, 15938446 bytes
>2018-07-11 05:10:09.677638 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809677611, "cf_name": "default", "job": 718, "event": "table_file_creation", "file_number": 10204, "file_size": 15938446, "table_properties": {"data_size": 15816487, "index_size": 69475, "filter_size": 51557, "raw_key_size": 67638, "raw_average_key_size": 20, "raw_value_size": 15719877, "raw_average_value_size": 4675, "num_data_blocks": 2035, "num_entries": 3362, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:10:09.719096 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 718] Compacted 1@0 + 1@1 files to L1 => 15938446 bytes
>2018-07-11 05:10:09.752628 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.752562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.2 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(2.1, 13.7) out(15.2), read-write-amplify(14.7) write-amplify(7.2) OK, records in: 3713, records dropped: 351
>
>2018-07-11 05:10:09.752637 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:10:09.752583) EVENT_LOG_v1 {"time_micros": 1531285809752574, "job": 718, "event": "compaction_finished", "compaction_time_micros": 232729, "output_level": 1, "num_output_files": 1, "total_output_size": 15938446, "num_input_records": 3713, "num_output_records": 3362, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:10:09.753140 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809753136, "job": 718, "event": "table_file_deletion", "file_number": 10203}
>2018-07-11 05:10:09.755807 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285809755804, "job": 718, "event": "table_file_deletion", "file_number": 10201}
>2018-07-11 05:10:09.755924 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:10:14.263918 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216192 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:14.412029 7f29149e4700  1 mon.magna102@0(leader).log v18947 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18947)
>2018-07-11 05:10:22.267379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216195 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:22.421050 7f29149e4700  1 mon.magna102@0(leader).log v18948 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18948)
>2018-07-11 05:10:28.269980 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216198 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:28.421628 7f29149e4700  1 mon.magna102@0(leader).log v18949 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18949)
>2018-07-11 05:10:36.273262 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216201 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:36.480939 7f29149e4700  1 mon.magna102@0(leader).log v18950 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18950)
>2018-07-11 05:10:44.276818 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216204 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:44.431662 7f29149e4700  1 mon.magna102@0(leader).log v18951 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18951)
>2018-07-11 05:10:49.327715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216207 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:49.416643 7f29149e4700  1 mon.magna102@0(leader).log v18952 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18952)
>2018-07-11 05:10:54.327947 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216210 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:54.450635 7f29149e4700  1 mon.magna102@0(leader).log v18953 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18953)
>2018-07-11 05:10:59.328140 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216213 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:10:59.417726 7f29149e4700  1 mon.magna102@0(leader).log v18954 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18954)
>2018-07-11 05:11:00.290264 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 05:11:04.426878 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10205. Immutable memtables: 0.
>
>2018-07-11 05:11:04.426922 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 719] Syncing log #10202
>2018-07-11 05:11:04.443436 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.426913) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:11:04.443447 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 719] Flushing memtable with next log file: 10205
>
>2018-07-11 05:11:04.443461 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864443453, "job": 719, "event": "flush_started", "num_memtables": 1, "num_entries": 516, "num_deletes": 251, "memory_usage": 590568}
>2018-07-11 05:11:04.443464 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 719] Level-0 flush table #10206: started
>2018-07-11 05:11:04.493967 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864493954, "cf_name": "default", "job": 719, "event": "table_file_creation", "file_number": 10206, "file_size": 565453, "table_properties": {"data_size": 559702, "index_size": 2421, "filter_size": 2411, "raw_key_size": 6911, "raw_average_key_size": 20, "raw_value_size": 553330, "raw_average_value_size": 1627, "num_data_blocks": 74, "num_entries": 340, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:11:04.493978 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 719] Level-0 flush table #10206: 565453 bytes OK
>2018-07-11 05:11:04.544010 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.510529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10206 started
>2018-07-11 05:11:04.544018 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.543974) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10206: memtable #1 done
>2018-07-11 05:11:04.544021 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.543989) EVENT_LOG_v1 {"time_micros": 1531285864543982, "job": 719, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:11:04.544023 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.544000) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:11:04.544029 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 719] Try to delete WAL files size 578397, prev total WAL file size 578397, number of live WAL files 2.
>
>2018-07-11 05:11:04.544237 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:11:04.544290 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.544283) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:11:04.544299 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 720] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:11:04.544305 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 719 Base level 0, inputs: [10206(552KB)], [10204(15MB)]
>
>2018-07-11 05:11:04.544318 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864544308, "job": 720, "event": "compaction_started", "files_L0": [10206], "files_L1": [10204], "score": -1, "input_data_size": 16503899}
>2018-07-11 05:11:04.719711 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 720] Generated table #10207: 3192 keys, 14506796 bytes
>2018-07-11 05:11:04.719760 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864719732, "cf_name": "default", "job": 720, "event": "table_file_creation", "file_number": 10207, "file_size": 14506796, "table_properties": {"data_size": 14394937, "index_size": 63732, "filter_size": 47200, "raw_key_size": 64302, "raw_average_key_size": 20, "raw_value_size": 14304703, "raw_average_value_size": 4481, "num_data_blocks": 1858, "num_entries": 3192, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:11:04.752865 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 720] Compacted 1@0 + 1@1 files to L1 => 14506796 bytes
>2018-07-11 05:11:04.786438 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.786371) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.1 rd, 69.6 wr, level 1, files in(1, 1) out(1) MB in(0.5, 15.2) out(13.8), read-write-amplify(54.8) write-amplify(25.7) OK, records in: 3702, records dropped: 510
>
>2018-07-11 05:11:04.786447 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:11:04.786393) EVENT_LOG_v1 {"time_micros": 1531285864786383, "job": 720, "event": "compaction_finished", "compaction_time_micros": 208530, "output_level": 1, "num_output_files": 1, "total_output_size": 14506796, "num_input_records": 3702, "num_output_records": 3192, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:11:04.786614 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864786610, "job": 720, "event": "table_file_deletion", "file_number": 10206}
>2018-07-11 05:11:04.789695 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285864789692, "job": 720, "event": "table_file_deletion", "file_number": 10204}
>2018-07-11 05:11:06.286374 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216216 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:06.470665 7f29149e4700  1 mon.magna102@0(leader).log v18955 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18955)
>2018-07-11 05:11:14.290089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216219 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:14.446395 7f29149e4700  1 mon.magna102@0(leader).log v18956 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18956)
>2018-07-11 05:11:19.329591 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216222 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:19.430156 7f29149e4700  1 mon.magna102@0(leader).log v18957 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18957)
>2018-07-11 05:11:28.296075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216225 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:28.447701 7f29149e4700  1 mon.magna102@0(leader).log v18958 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18958)
>2018-07-11 05:11:34.340844 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216228 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:34.474536 7f29149e4700  1 mon.magna102@0(leader).log v18959 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18959)
>2018-07-11 05:11:40.301089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216231 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:40.475118 7f29149e4700  1 mon.magna102@0(leader).log v18960 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18960)
>2018-07-11 05:11:48.304671 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216234 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:48.484128 7f29149e4700  1 mon.magna102@0(leader).log v18961 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18961)
>2018-07-11 05:11:54.351047 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216237 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:54.476414 7f29149e4700  1 mon.magna102@0(leader).log v18962 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18962)
>2018-07-11 05:11:59.351279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216240 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:11:59.443498 7f29149e4700  1 mon.magna102@0(leader).log v18963 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18963)
>2018-07-11 05:12:00.290444 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:12:04.352125 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216243 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:04.469172 7f29149e4700  1 mon.magna102@0(leader).log v18964 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18964)
>2018-07-11 05:12:09.352332 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216246 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:09.452918 7f29149e4700  1 mon.magna102@0(leader).log v18965 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18965)
>2018-07-11 05:12:18.317423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216249 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:18.470463 7f29149e4700  1 mon.magna102@0(leader).log v18966 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18966)
>2018-07-11 05:12:24.320124 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216252 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:24.471100 7f29149e4700  1 mon.magna102@0(leader).log v18967 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18967)
>2018-07-11 05:12:29.354386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216255 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:29.504876 7f29149e4700  1 mon.magna102@0(leader).log v18968 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18968)
>2018-07-11 05:12:39.356575 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216258 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:39.465320 7f29149e4700  1 mon.magna102@0(leader).log v18969 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18969)
>2018-07-11 05:12:44.365569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216261 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:44.482599 7f29149e4700  1 mon.magna102@0(leader).log v18970 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18970)
>2018-07-11 05:12:49.365786 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216264 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:49.466393 7f29149e4700  1 mon.magna102@0(leader).log v18971 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18971)
>2018-07-11 05:12:58.334909 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216270 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:12:58.492442 7f29149e4700  1 mon.magna102@0(leader).log v18972 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18972)
>2018-07-11 05:13:00.290583 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:13:05.492916 7f29149e4700  1 mon.magna102@0(leader).log v18973 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18973)
>2018-07-11 05:13:06.492969 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216273 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:06.668326 7f29149e4700  1 mon.magna102@0(leader).log v18974 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18974)
>2018-07-11 05:13:14.341785 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216276 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:14.510699 7f29149e4700  1 mon.magna102@0(leader).log v18975 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18975)
>2018-07-11 05:13:19.377216 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216279 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:19.494464 7f29149e4700  1 mon.magna102@0(leader).log v18976 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18976)
>2018-07-11 05:13:29.386581 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216282 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:29.513220 7f29149e4700  1 mon.magna102@0(leader).log v18977 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18977)
>2018-07-11 05:13:34.386820 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216285 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:34.505626 7f29149e4700  1 mon.magna102@0(leader).log v18978 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18978)
>2018-07-11 05:13:39.387015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216288 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:39.481064 7f29149e4700  1 mon.magna102@0(leader).log v18979 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18979)
>2018-07-11 05:13:44.389673 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216291 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:44.506715 7f29149e4700  1 mon.magna102@0(leader).log v18980 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18980)
>2018-07-11 05:13:45.573516 7f29149e4700  1 mon.magna102@0(leader).log v18981 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 18981)
>2018-07-11 05:13:52.358240 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216294 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:52.515940 7f29149e4700  1 mon.magna102@0(leader).log v18982 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18982)
>2018-07-11 05:13:59.391132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216297 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:13:59.491486 7f29149e4700  1 mon.magna102@0(leader).log v18983 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18983)
>2018-07-11 05:14:00.290756 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:14:08.365052 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216303 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:08.525730 7f29149e4700  1 mon.magna102@0(leader).log v18984 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18984)
>2018-07-11 05:14:18.369299 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216306 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:18.518229 7f29149e4700  1 mon.magna102@0(leader).log v18985 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18985)
>2018-07-11 05:14:24.371952 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216309 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:24.527106 7f29149e4700  1 mon.magna102@0(leader).log v18986 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18986)
>2018-07-11 05:14:29.410292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216312 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:29.512117 7f29149e4700  1 mon.magna102@0(leader).log v18987 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18987)
>2018-07-11 05:14:34.412399 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216315 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:34.529468 7f29149e4700  1 mon.magna102@0(leader).log v18988 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18988)
>2018-07-11 05:14:39.412620 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216318 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:39.521511 7f29149e4700  1 mon.magna102@0(leader).log v18989 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18989)
>2018-07-11 05:14:49.422177 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216321 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:49.555872 7f29149e4700  1 mon.magna102@0(leader).log v18990 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18990)
>2018-07-11 05:14:58.386589 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216327 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:14:58.556748 7f29149e4700  1 mon.magna102@0(leader).log v18991 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18991)
>2018-07-11 05:15:00.290946 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:15:06.390030 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216330 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:06.540664 7f29149e4700  1 mon.magna102@0(leader).log v18992 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18992)
>2018-07-11 05:15:14.393465 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216336 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:14.549814 7f29149e4700  1 mon.magna102@0(leader).log v18993 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18993)
>2018-07-11 05:15:24.397741 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216339 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:24.558969 7f29149e4700  1 mon.magna102@0(leader).log v18994 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18994)
>2018-07-11 05:15:30.400263 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216342 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:30.552433 7f29149e4700  1 mon.magna102@0(leader).log v18995 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18995)
>2018-07-11 05:15:36.402901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216345 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:36.553016 7f29149e4700  1 mon.magna102@0(leader).log v18996 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18996)
>2018-07-11 05:15:44.436715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216348 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:44.553759 7f29149e4700  1 mon.magna102@0(leader).log v18997 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18997)
>2018-07-11 05:15:49.436948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216351 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:49.545857 7f29149e4700  1 mon.magna102@0(leader).log v18998 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18998)
>2018-07-11 05:15:59.446320 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216354 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:15:59.555125 7f29149e4700  1 mon.magna102@0(leader).log v18999 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 18999)
>2018-07-11 05:16:00.291120 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:16:04.447062 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216357 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:04.564107 7f29149e4700  1 mon.magna102@0(leader).log v19000 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19000)
>2018-07-11 05:16:09.447252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216360 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:09.556213 7f29149e4700  1 mon.magna102@0(leader).log v19001 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19001)
>2018-07-11 05:16:14.456504 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216363 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:14.573524 7f29149e4700  1 mon.magna102@0(leader).log v19002 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19002)
>2018-07-11 05:16:19.456701 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216366 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:19.557296 7f29149e4700  1 mon.magna102@0(leader).log v19003 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19003)
>2018-07-11 05:16:29.457729 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216369 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:29.584372 7f29149e4700  1 mon.magna102@0(leader).log v19004 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19004)
>2018-07-11 05:16:34.468084 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216372 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:34.601771 7f29149e4700  1 mon.magna102@0(leader).log v19005 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19005)
>2018-07-11 05:16:39.468285 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216375 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:39.593856 7f29149e4700  1 mon.magna102@0(leader).log v19006 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19006)
>2018-07-11 05:16:44.468515 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216378 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:44.594516 7f29149e4700  1 mon.magna102@0(leader).log v19007 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19007)
>2018-07-11 05:16:52.435716 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216381 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:52.586879 7f29149e4700  1 mon.magna102@0(leader).log v19008 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19008)
>2018-07-11 05:16:59.470404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216384 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:16:59.570644 7f29149e4700  1 mon.magna102@0(leader).log v19009 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19009)
>2018-07-11 05:17:00.291313 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:17:09.471324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216387 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:09.580053 7f29149e4700  1 mon.magna102@0(leader).log v19010 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19010)
>2018-07-11 05:17:14.480352 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216390 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:14.597370 7f29149e4700  1 mon.magna102@0(leader).log v19011 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19011)
>2018-07-11 05:17:19.480562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216393 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:19.581128 7f29149e4700  1 mon.magna102@0(leader).log v19012 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19012)
>2018-07-11 05:17:24.481440 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216396 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:24.599619 7f29149e4700  1 mon.magna102@0(leader).log v19013 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19013)
>2018-07-11 05:17:26.616867 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10208. Immutable memtables: 0.
>
>2018-07-11 05:17:26.616911 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 721] Syncing log #10205
>2018-07-11 05:17:26.633339 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.616902) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:17:26.633351 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 721] Flushing memtable with next log file: 10208
>
>2018-07-11 05:17:26.633377 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286246633369, "job": 721, "event": "flush_started", "num_memtables": 1, "num_entries": 2065, "num_deletes": 251, "memory_usage": 3987816}
>2018-07-11 05:17:26.633380 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 721] Level-0 flush table #10209: started
>2018-07-11 05:17:26.708975 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286246708949, "cf_name": "default", "job": 721, "event": "table_file_creation", "file_number": 10209, "file_size": 3846729, "table_properties": {"data_size": 3816194, "index_size": 16849, "filter_size": 12760, "raw_key_size": 16868, "raw_average_key_size": 20, "raw_value_size": 3792745, "raw_average_value_size": 4630, "num_data_blocks": 505, "num_entries": 819, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:17:26.708988 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 721] Level-0 flush table #10209: 3846729 bytes OK
>2018-07-11 05:17:26.758990 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.725483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10209 started
>2018-07-11 05:17:26.758998 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.758949) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10209: memtable #1 done
>2018-07-11 05:17:26.759000 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.758964) EVENT_LOG_v1 {"time_micros": 1531286246758958, "job": 721, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:17:26.759002 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.758978) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:17:26.759022 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 721] Try to delete WAL files size 3946736, prev total WAL file size 3946736, number of live WAL files 2.
>
>2018-07-11 05:17:26.759992 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:17:26.760060 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:26.760050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:17:26.760071 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 722] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:17:26.760077 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 721 Base level 0, inputs: [10209(3756KB)], [10207(13MB)]
>
>2018-07-11 05:17:26.760090 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286246760080, "job": 722, "event": "compaction_started", "files_L0": [10209], "files_L1": [10207], "score": -1, "input_data_size": 18353525}
>2018-07-11 05:17:26.951338 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 722] Generated table #10210: 3501 keys, 16350462 bytes
>2018-07-11 05:17:26.951388 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286246951371, "cf_name": "default", "job": 722, "event": "table_file_creation", "file_number": 10210, "file_size": 16350462, "table_properties": {"data_size": 16223761, "index_size": 72601, "filter_size": 53173, "raw_key_size": 70923, "raw_average_key_size": 20, "raw_value_size": 16122840, "raw_average_value_size": 4605, "num_data_blocks": 2111, "num_entries": 3501, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:17:26.984520 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 722] Compacted 1@0 + 1@1 files to L1 => 16350462 bytes
>2018-07-11 05:17:27.018046 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:27.017979) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.8 rd, 72.9 wr, level 1, files in(1, 1) out(1) MB in(3.7, 13.8) out(15.6), read-write-amplify(9.0) write-amplify(4.3) OK, records in: 4011, records dropped: 510
>
>2018-07-11 05:17:27.018055 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:27.018001) EVENT_LOG_v1 {"time_micros": 1531286247017991, "job": 722, "event": "compaction_finished", "compaction_time_micros": 224415, "output_level": 1, "num_output_files": 1, "total_output_size": 16350462, "num_input_records": 4011, "num_output_records": 3501, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:17:27.018892 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286247018888, "job": 722, "event": "table_file_deletion", "file_number": 10209}
>2018-07-11 05:17:27.021613 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286247021610, "job": 722, "event": "table_file_deletion", "file_number": 10207}
>2018-07-11 05:17:29.481663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216399 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:29.593828 7f29149e4700  1 mon.magna102@0(leader).log v19014 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19014)
>2018-07-11 05:17:39.485970 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216402 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:39.586366 7f29149e4700  1 mon.magna102@0(leader).log v19015 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19015)
>2018-07-11 05:17:44.495067 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216405 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:44.595881 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10211. Immutable memtables: 0.
>
>2018-07-11 05:17:44.628956 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.595939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:17:44.628971 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 723] Flushing memtable with next log file: 10211
>
>2018-07-11 05:17:44.628985 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286264628978, "job": 723, "event": "flush_started", "num_memtables": 1, "num_entries": 584, "num_deletes": 500, "memory_usage": 211336}
>2018-07-11 05:17:44.628988 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 723] Level-0 flush table #10212: started
>2018-07-11 05:17:44.677558 7f29149e4700  1 mon.magna102@0(leader).log v19016 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19016)
>2018-07-11 05:17:44.696242 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286264696228, "cf_name": "default", "job": 723, "event": "table_file_creation", "file_number": 10212, "file_size": 193037, "table_properties": {"data_size": 189467, "index_size": 802, "filter_size": 1849, "raw_key_size": 10976, "raw_average_key_size": 20, "raw_value_size": 181802, "raw_average_value_size": 341, "num_data_blocks": 25, "num_entries": 533, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 05:17:44.696253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 723] Level-0 flush table #10212: 193037 bytes OK
>2018-07-11 05:17:44.754601 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.712791) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10212 started
>2018-07-11 05:17:44.754609 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.754565) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10212: memtable #1 done
>2018-07-11 05:17:44.754611 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.754580) EVENT_LOG_v1 {"time_micros": 1531286264754573, "job": 723, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:17:44.754613 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.754591) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:17:44.754619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 723] Try to delete WAL files size 195565, prev total WAL file size 213999, number of live WAL files 2.
>
>2018-07-11 05:17:44.754743 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:17:44.754794 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.754787) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 05:17:44.754803 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 724] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:17:44.754809 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 723 Base level 0, inputs: [10212(188KB)], [10210(15MB)]
>
>2018-07-11 05:17:44.754822 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286264754812, "job": 724, "event": "compaction_started", "files_L0": [10212], "files_L1": [10210], "score": -1, "input_data_size": 16543499}
>2018-07-11 05:17:44.930323 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 724] Generated table #10213: 3026 keys, 14258723 bytes
>2018-07-11 05:17:44.930361 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286264930346, "cf_name": "default", "job": 724, "event": "table_file_creation", "file_number": 10213, "file_size": 14258723, "table_properties": {"data_size": 14147341, "index_size": 64015, "filter_size": 46440, "raw_key_size": 61193, "raw_average_key_size": 20, "raw_value_size": 14059238, "raw_average_value_size": 4646, "num_data_blocks": 1871, "num_entries": 3026, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:17:44.963509 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 724] Compacted 1@0 + 1@1 files to L1 => 14258723 bytes
>2018-07-11 05:17:44.997017 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.996948) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.3 rd, 68.3 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.6) out(13.6), read-write-amplify(159.6) write-amplify(73.9) OK, records in: 4034, records dropped: 1008
>
>2018-07-11 05:17:44.997026 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:44.996969) EVENT_LOG_v1 {"time_micros": 1531286264996960, "job": 724, "event": "compaction_finished", "compaction_time_micros": 208670, "output_level": 1, "num_output_files": 1, "total_output_size": 14258723, "num_input_records": 4034, "num_output_records": 3026, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:17:44.997132 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286264997128, "job": 724, "event": "table_file_deletion", "file_number": 10212}
>2018-07-11 05:17:45.000158 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265000155, "job": 724, "event": "table_file_deletion", "file_number": 10210}
>2018-07-11 05:17:45.000416 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10214. Immutable memtables: 0.
>
>2018-07-11 05:17:45.000457 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 725] Syncing log #10211
>2018-07-11 05:17:45.013947 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.000449) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:17:45.013958 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 725] Flushing memtable with next log file: 10214
>
>2018-07-11 05:17:45.013972 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265013965, "job": 725, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20776}
>2018-07-11 05:17:45.013975 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 725] Level-0 flush table #10215: started
>2018-07-11 05:17:45.056082 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265056069, "cf_name": "default", "job": 725, "event": "table_file_creation", "file_number": 10215, "file_size": 19549, "table_properties": {"data_size": 18483, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18260, "raw_average_value_size": 2282, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:17:45.056092 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 725] Level-0 flush table #10215: 19549 bytes OK
>2018-07-11 05:17:45.106136 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.072682) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10215 started
>2018-07-11 05:17:45.106144 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.106102) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10215: memtable #1 done
>2018-07-11 05:17:45.106147 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.106116) EVENT_LOG_v1 {"time_micros": 1531286265106110, "job": 725, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:17:45.106149 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.106128) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:17:45.106153 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 725] Try to delete WAL files size 18434, prev total WAL file size 18434, number of live WAL files 2.
>
>2018-07-11 05:17:45.106252 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:17:45.106300 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.106294) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 05:17:45.106308 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 726] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:17:45.106314 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 725 Base level 0, inputs: [10215(19KB)], [10213(13MB)]
>
>2018-07-11 05:17:45.106326 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265106317, "job": 726, "event": "compaction_started", "files_L0": [10215], "files_L1": [10213], "score": -1, "input_data_size": 14278272}
>2018-07-11 05:17:45.281853 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 726] Generated table #10216: 3029 keys, 14277161 bytes
>2018-07-11 05:17:45.281897 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265281870, "cf_name": "default", "job": 726, "event": "table_file_creation", "file_number": 10216, "file_size": 14277161, "table_properties": {"data_size": 14165655, "index_size": 64083, "filter_size": 46496, "raw_key_size": 61254, "raw_average_key_size": 20, "raw_value_size": 14077458, "raw_average_value_size": 4647, "num_data_blocks": 1873, "num_entries": 3029, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:17:45.315049 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 726] Compacted 1@0 + 1@1 files to L1 => 14277161 bytes
>2018-07-11 05:17:45.349809 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.349740) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.4 rd, 68.4 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.6) out(13.6), read-write-amplify(1460.7) write-amplify(730.3) OK, records in: 3034, records dropped: 5
>
>2018-07-11 05:17:45.349818 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:17:45.349761) EVENT_LOG_v1 {"time_micros": 1531286265349752, "job": 726, "event": "compaction_finished", "compaction_time_micros": 208706, "output_level": 1, "num_output_files": 1, "total_output_size": 14277161, "num_input_records": 3034, "num_output_records": 3029, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:17:45.349894 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265349890, "job": 726, "event": "table_file_deletion", "file_number": 10215}
>2018-07-11 05:17:45.352597 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286265352594, "job": 726, "event": "table_file_deletion", "file_number": 10213}
>2018-07-11 05:17:49.499487 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216408 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:49.617380 7f29149e4700  1 mon.magna102@0(leader).log v19017 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19017)
>2018-07-11 05:17:54.500984 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216411 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:54.634668 7f29149e4700  1 mon.magna102@0(leader).log v19018 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19018)
>2018-07-11 05:17:59.501219 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216414 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:17:59.635099 7f29149e4700  1 mon.magna102@0(leader).log v19019 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19019)
>2018-07-11 05:18:00.291492 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:18:08.469141 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216417 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:08.644317 7f29149e4700  1 mon.magna102@0(leader).log v19020 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19020)
>2018-07-11 05:18:14.511220 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216420 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:14.644914 7f29149e4700  1 mon.magna102@0(leader).log v19021 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19021)
>2018-07-11 05:18:19.511434 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216423 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:19.620524 7f29149e4700  1 mon.magna102@0(leader).log v19022 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19022)
>2018-07-11 05:18:22.437411 7f29149e4700  1 mon.magna102@0(leader).log v19023 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19023)
>2018-07-11 05:18:29.511857 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216426 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:29.580076 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10217. Immutable memtables: 0.
>
>2018-07-11 05:18:29.580118 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 727] Syncing log #10214
>2018-07-11 05:18:29.596641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.580108) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:18:29.596653 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 727] Flushing memtable with next log file: 10217
>
>2018-07-11 05:18:29.596679 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309596671, "job": 727, "event": "flush_started", "num_memtables": 1, "num_entries": 465, "num_deletes": 250, "memory_usage": 480496}
>2018-07-11 05:18:29.596689 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 727] Level-0 flush table #10218: started
>2018-07-11 05:18:29.664044 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309664029, "cf_name": "default", "job": 727, "event": "table_file_creation", "file_number": 10218, "file_size": 350184, "table_properties": {"data_size": 346133, "index_size": 1415, "filter_size": 1717, "raw_key_size": 6728, "raw_average_key_size": 21, "raw_value_size": 340918, "raw_average_value_size": 1106, "num_data_blocks": 44, "num_entries": 308, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:18:29.664054 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 727] Level-0 flush table #10218: 350184 bytes OK
>2018-07-11 05:18:29.697466 7f29149e4700  1 mon.magna102@0(leader).log v19024 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19024)
>2018-07-11 05:18:29.730736 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.697238) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10218 started
>2018-07-11 05:18:29.730744 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.730700) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10218: memtable #1 done
>2018-07-11 05:18:29.730746 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.730715) EVENT_LOG_v1 {"time_micros": 1531286309730709, "job": 727, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:18:29.730761 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.730726) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:18:29.730768 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 727] Try to delete WAL files size 469647, prev total WAL file size 487941, number of live WAL files 2.
>
>2018-07-11 05:18:29.730961 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:18:29.731010 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.731003) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:18:29.731020 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 728] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:18:29.731026 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 727 Base level 0, inputs: [10218(341KB)], [10216(13MB)]
>
>2018-07-11 05:18:29.731039 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309731029, "job": 728, "event": "compaction_started", "files_L0": [10218], "files_L1": [10216], "score": -1, "input_data_size": 14627345}
>2018-07-11 05:18:29.889763 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 728] Generated table #10219: 2845 keys, 13023373 bytes
>2018-07-11 05:18:29.889812 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309889784, "cf_name": "default", "job": 728, "event": "table_file_creation", "file_number": 10219, "file_size": 13023373, "table_properties": {"data_size": 12922682, "index_size": 57396, "filter_size": 42368, "raw_key_size": 57125, "raw_average_key_size": 20, "raw_value_size": 12841873, "raw_average_value_size": 4513, "num_data_blocks": 1683, "num_entries": 2845, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:18:29.924188 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 728] Compacted 1@0 + 1@1 files to L1 => 13023373 bytes
>2018-07-11 05:18:29.957724 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.957655) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.7 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(0.3, 13.6) out(12.4), read-write-amplify(79.0) write-amplify(37.2) OK, records in: 3337, records dropped: 492
>
>2018-07-11 05:18:29.957732 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.957677) EVENT_LOG_v1 {"time_micros": 1531286309957667, "job": 728, "event": "compaction_finished", "compaction_time_micros": 193126, "output_level": 1, "num_output_files": 1, "total_output_size": 13023373, "num_input_records": 3337, "num_output_records": 2845, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:18:29.957880 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309957876, "job": 728, "event": "table_file_deletion", "file_number": 10218}
>2018-07-11 05:18:29.960603 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309960600, "job": 728, "event": "table_file_deletion", "file_number": 10216}
>2018-07-11 05:18:29.960924 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10220. Immutable memtables: 0.
>
>2018-07-11 05:18:29.960964 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 729] Syncing log #10217
>2018-07-11 05:18:29.974559 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:29.960956) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:18:29.974568 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 729] Flushing memtable with next log file: 10220
>
>2018-07-11 05:18:29.974580 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286309974574, "job": 729, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20648}
>2018-07-11 05:18:29.974582 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 729] Level-0 flush table #10221: started
>2018-07-11 05:18:30.008483 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286310008470, "cf_name": "default", "job": 729, "event": "table_file_creation", "file_number": 10221, "file_size": 19409, "table_properties": {"data_size": 18343, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18120, "raw_average_value_size": 2265, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:18:30.008493 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 729] Level-0 flush table #10221: 19409 bytes OK
>2018-07-11 05:18:30.058512 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.025043) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10221 started
>2018-07-11 05:18:30.058520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.058478) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10221: memtable #1 done
>2018-07-11 05:18:30.058523 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.058493) EVENT_LOG_v1 {"time_micros": 1531286310058487, "job": 729, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:18:30.058538 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.058504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:18:30.058543 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 729] Try to delete WAL files size 18294, prev total WAL file size 18294, number of live WAL files 2.
>
>2018-07-11 05:18:30.058641 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:18:30.058688 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.058682) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:18:30.058697 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 730] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:18:30.058703 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 729 Base level 0, inputs: [10221(18KB)], [10219(12MB)]
>
>2018-07-11 05:18:30.058716 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286310058706, "job": 730, "event": "compaction_started", "files_L0": [10221], "files_L1": [10219], "score": -1, "input_data_size": 13042782}
>2018-07-11 05:18:30.217579 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 730] Generated table #10222: 2848 keys, 13041669 bytes
>2018-07-11 05:18:30.217622 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286310217596, "cf_name": "default", "job": 730, "event": "table_file_creation", "file_number": 10222, "file_size": 13041669, "table_properties": {"data_size": 12940852, "index_size": 57465, "filter_size": 42425, "raw_key_size": 57186, "raw_average_key_size": 20, "raw_value_size": 12859953, "raw_average_value_size": 4515, "num_data_blocks": 1685, "num_entries": 2848, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:18:30.250764 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 730] Compacted 1@0 + 1@1 files to L1 => 13041669 bytes
>2018-07-11 05:18:30.292627 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.292562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 67.9 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 12.4) out(12.4), read-write-amplify(1343.9) write-amplify(671.9) OK, records in: 2853, records dropped: 5
>
>2018-07-11 05:18:30.292636 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:18:30.292584) EVENT_LOG_v1 {"time_micros": 1531286310292574, "job": 730, "event": "compaction_finished", "compaction_time_micros": 192031, "output_level": 1, "num_output_files": 1, "total_output_size": 13041669, "num_input_records": 2853, "num_output_records": 2848, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:18:30.292729 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286310292725, "job": 730, "event": "table_file_deletion", "file_number": 10221}
>2018-07-11 05:18:30.295264 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286310295261, "job": 730, "event": "table_file_deletion", "file_number": 10219}
>2018-07-11 05:18:32.459960 7f29149e4700  1 mon.magna102@0(leader).log v19025 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19025)
>2018-07-11 05:18:39.546854 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216429 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:39.644129 7f29149e4700  1 mon.magna102@0(leader).log v19026 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19026)
>2018-07-11 05:18:44.547099 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216432 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:44.678102 7f29149e4700  1 mon.magna102@0(leader).log v19027 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19027)
>2018-07-11 05:18:49.547268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216435 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:49.645316 7f29149e4700  1 mon.magna102@0(leader).log v19028 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19028)
>2018-07-11 05:18:54.547510 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216438 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:54.662803 7f29149e4700  1 mon.magna102@0(leader).log v19029 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19029)
>2018-07-11 05:18:59.547710 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216441 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:18:59.621554 7f29149e4700  1 mon.magna102@0(leader).log v19030 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19030)
>2018-07-11 05:19:00.291682 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:19:06.494300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216444 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:06.647332 7f29149e4700  1 mon.magna102@0(leader).log v19031 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19031)
>2018-07-11 05:19:14.497830 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216447 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:14.649165 7f29149e4700  1 mon.magna102@0(leader).log v19032 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19032)
>2018-07-11 05:19:20.500265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216450 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:20.649819 7f29149e4700  1 mon.magna102@0(leader).log v19033 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19033)
>2018-07-11 05:19:26.502939 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216453 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:26.650384 7f29149e4700  1 mon.magna102@0(leader).log v19034 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19034)
>2018-07-11 05:19:34.548935 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216456 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:34.667649 7f29149e4700  1 mon.magna102@0(leader).log v19035 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19035)
>2018-07-11 05:19:39.549136 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216459 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:39.643237 7f29149e4700  1 mon.magna102@0(leader).log v19036 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19036)
>2018-07-11 05:19:44.549379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216462 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:44.668872 7f29149e4700  1 mon.magna102@0(leader).log v19037 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19037)
>2018-07-11 05:19:45.735654 7f29149e4700  1 mon.magna102@0(leader).log v19038 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19038)
>2018-07-11 05:19:52.514198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216465 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:52.661620 7f29149e4700  1 mon.magna102@0(leader).log v19039 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19039)
>2018-07-11 05:19:59.586356 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216468 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:19:59.703856 7f29149e4700  1 mon.magna102@0(leader).log v19040 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19040)
>2018-07-11 05:20:00.291858 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:20:04.586592 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216471 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:04.714392 7f29149e4700  1 mon.magna102@0(leader).log v19041 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19041)
>2018-07-11 05:20:10.521932 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216474 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:10.706591 7f29149e4700  1 mon.magna102@0(leader).log v19042 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19042)
>2018-07-11 05:20:16.524444 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216477 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:16.698810 7f29149e4700  1 mon.magna102@0(leader).log v19043 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19043)
>2018-07-11 05:20:22.527148 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216480 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:22.682745 7f29149e4700  1 mon.magna102@0(leader).log v19044 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19044)
>2018-07-11 05:20:28.529652 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216483 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:28.675003 7f29149e4700  1 mon.magna102@0(leader).log v19045 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19045)
>2018-07-11 05:20:34.532213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216486 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:34.684210 7f29149e4700  1 mon.magna102@0(leader).log v19046 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19046)
>2018-07-11 05:20:40.534769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216489 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:40.686028 7f29149e4700  1 mon.magna102@0(leader).log v19047 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19047)
>2018-07-11 05:20:48.538261 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216492 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:48.686749 7f29149e4700  1 mon.magna102@0(leader).log v19048 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19048)
>2018-07-11 05:20:54.588394 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216495 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:54.704057 7f29149e4700  1 mon.magna102@0(leader).log v19049 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19049)
>2018-07-11 05:20:59.588588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216498 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:20:59.679487 7f29149e4700  1 mon.magna102@0(leader).log v19050 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19050)
>2018-07-11 05:21:00.292045 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:21:04.588814 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216501 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:04.713479 7f29149e4700  1 mon.magna102@0(leader).log v19051 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19051)
>2018-07-11 05:21:14.549364 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216504 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:14.706121 7f29149e4700  1 mon.magna102@0(leader).log v19052 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19052)
>2018-07-11 05:21:19.589324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216507 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:19.698236 7f29149e4700  1 mon.magna102@0(leader).log v19053 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19053)
>2018-07-11 05:21:28.555439 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216513 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:28.716002 7f29149e4700  1 mon.magna102@0(leader).log v19054 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19054)
>2018-07-11 05:21:38.559610 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216516 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:38.743086 7f29149e4700  1 mon.magna102@0(leader).log v19055 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19055)
>2018-07-11 05:21:44.618346 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216519 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:44.743699 7f29149e4700  1 mon.magna102@0(leader).log v19056 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19056)
>2018-07-11 05:21:49.618532 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216522 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:49.694153 7f29149e4700  1 mon.magna102@0(leader).log v19057 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19057)
>2018-07-11 05:21:54.618758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216525 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:54.736489 7f29149e4700  1 mon.magna102@0(leader).log v19058 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19058)
>2018-07-11 05:21:59.618957 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216528 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:21:59.703602 7f29149e4700  1 mon.magna102@0(leader).log v19059 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19059)
>2018-07-11 05:22:00.292224 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:22:06.571456 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216531 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:06.721025 7f29149e4700  1 mon.magna102@0(leader).log v19060 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19060)
>2018-07-11 05:22:12.574233 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216534 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:12.721647 7f29149e4700  1 mon.magna102@0(leader).log v19061 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19061)
>2018-07-11 05:22:18.576666 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216537 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:18.722234 7f29149e4700  1 mon.magna102@0(leader).log v19062 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19062)
>2018-07-11 05:22:24.579283 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216540 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:24.731141 7f29149e4700  1 mon.magna102@0(leader).log v19063 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19063)
>2018-07-11 05:22:34.583499 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216543 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:34.733265 7f29149e4700  1 mon.magna102@0(leader).log v19064 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19064)
>2018-07-11 05:22:42.586908 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216546 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:42.742398 7f29149e4700  1 mon.magna102@0(leader).log v19065 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19065)
>2018-07-11 05:22:49.625769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216549 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:49.726344 7f29149e4700  1 mon.magna102@0(leader).log v19066 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19066)
>2018-07-11 05:22:54.626967 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216552 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:54.760439 7f29149e4700  1 mon.magna102@0(leader).log v19067 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19067)
>2018-07-11 05:22:59.627168 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216555 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:22:59.760844 7f29149e4700  1 mon.magna102@0(leader).log v19068 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19068)
>2018-07-11 05:23:00.292401 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:23:04.636173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216558 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:04.769854 7f29149e4700  1 mon.magna102@0(leader).log v19069 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19069)
>2018-07-11 05:23:09.636397 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216561 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:09.753611 7f29149e4700  1 mon.magna102@0(leader).log v19070 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19070)
>2018-07-11 05:23:18.602202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216564 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:18.754572 7f29149e4700  1 mon.magna102@0(leader).log v19071 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19071)
>2018-07-11 05:23:24.638188 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216567 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:24.755156 7f29149e4700  1 mon.magna102@0(leader).log v19072 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19072)
>2018-07-11 05:23:29.638371 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216570 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:29.730645 7f29149e4700  1 mon.magna102@0(leader).log v19073 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19073)
>2018-07-11 05:23:34.640376 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216573 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:34.757532 7f29149e4700  1 mon.magna102@0(leader).log v19074 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19074)
>2018-07-11 05:23:38.741540 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10223. Immutable memtables: 0.
>
>2018-07-11 05:23:38.741583 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 731] Syncing log #10220
>2018-07-11 05:23:38.758088 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.741573) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:23:38.758099 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 731] Flushing memtable with next log file: 10223
>
>2018-07-11 05:23:38.758126 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286618758118, "job": 731, "event": "flush_started", "num_memtables": 1, "num_entries": 1750, "num_deletes": 251, "memory_usage": 3265664}
>2018-07-11 05:23:38.758129 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 731] Level-0 flush table #10224: started
>2018-07-11 05:23:38.825314 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286618825299, "cf_name": "default", "job": 731, "event": "table_file_creation", "file_number": 10224, "file_size": 3149723, "table_properties": {"data_size": 3124374, "index_size": 13834, "filter_size": 10591, "raw_key_size": 14942, "raw_average_key_size": 20, "raw_value_size": 3104332, "raw_average_value_size": 4275, "num_data_blocks": 416, "num_entries": 726, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:23:38.825325 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 731] Level-0 flush table #10224: 3149723 bytes OK
>2018-07-11 05:23:38.875326 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.841806) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10224 started
>2018-07-11 05:23:38.875334 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.875288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10224: memtable #1 done
>2018-07-11 05:23:38.875336 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.875303) EVENT_LOG_v1 {"time_micros": 1531286618875296, "job": 731, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:23:38.875352 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.875316) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:23:38.875358 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 731] Try to delete WAL files size 3230550, prev total WAL file size 3230550, number of live WAL files 2.
>
>2018-07-11 05:23:38.876173 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:23:38.876379 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 05:23:38.876386 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 48645.7 total, 809.4 interval
>Cumulative writes: 68K writes, 327K keys, 68K commit groups, 1.0 writes per commit group, ingest: 0.55 GB, 0.01 MB/s
>Cumulative WAL: 68K writes, 68K syncs, 1.00 writes per sync, written: 0.55 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1078 writes, 5396 keys, 1078 commit groups, 1.0 writes per commit group, ingest: 8.07 MB, 0.01 MB/s
>Interval WAL: 1078 writes, 1078 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.00 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        30       366    0.083       0      0
>  L1      1/1   12.44 MB   0.0      5.7     0.5      5.2       5.2     -0.0       0.0  10.9     72.3     66.2        81       364    0.222   1287K   151K
> Sum      2/2   15.44 MB   0.0      5.7     0.5      5.2       5.7      0.5       0.0  11.9     52.5     52.5       111       730    0.152   1287K   151K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  13.5     52.3     52.1         2        14    0.143     24K   2881
>Uptime(secs): 48645.7 total, 48645.7 interval
>Flush(GB): cumulative 0.481, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.71 GB write, 0.12 MB/s write, 5.71 GB read, 0.12 MB/s read, 111.2 seconds
>Interval compaction: 0.10 GB write, 0.00 MB/s write, 0.10 GB read, 0.00 MB/s read, 2.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.00 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        30       366    0.083       0      0
>  L1      1/1   12.44 MB   0.0      5.7     0.5      5.2       5.2     -0.0       0.0  10.9     72.3     66.2        81       364    0.222   1287K   151K
> Sum      2/2   15.44 MB   0.0      5.7     0.5      5.2       5.7      0.5       0.0  11.9     52.5     52.5       111       730    0.152   1287K   151K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 48645.7 total, 48645.7 interval
>Flush(GB): cumulative 0.481, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.71 GB write, 0.12 MB/s write, 5.71 GB read, 0.12 MB/s read, 111.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 05:23:38.876407 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:23:38.876400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:23:38.876412 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 732] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:23:38.876417 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 731 Base level 0, inputs: [10224(3075KB)], [10222(12MB)]
>
>2018-07-11 05:23:38.876431 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286618876421, "job": 732, "event": "compaction_started", "files_L0": [10224], "files_L1": [10222], "score": -1, "input_data_size": 16191392}
>2018-07-11 05:23:39.059367 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 732] Generated table #10225: 3061 keys, 14190663 bytes
>2018-07-11 05:23:39.059413 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286619059397, "cf_name": "default", "job": 732, "event": "table_file_creation", "file_number": 10225, "file_size": 14190663, "table_properties": {"data_size": 14080183, "index_size": 63316, "filter_size": 46237, "raw_key_size": 61803, "raw_average_key_size": 20, "raw_value_size": 13992078, "raw_average_value_size": 4571, "num_data_blocks": 1849, "num_entries": 3061, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:23:39.092556 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 732] Compacted 1@0 + 1@1 files to L1 => 14190663 bytes
>2018-07-11 05:23:39.126064 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:23:39.125997) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.9 rd, 65.7 wr, level 1, files in(1, 1) out(1) MB in(3.0, 12.4) out(13.5), read-write-amplify(9.6) write-amplify(4.5) OK, records in: 3574, records dropped: 513
>
>2018-07-11 05:23:39.126073 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:23:39.126018) EVENT_LOG_v1 {"time_micros": 1531286619126009, "job": 732, "event": "compaction_finished", "compaction_time_micros": 216108, "output_level": 1, "num_output_files": 1, "total_output_size": 14190663, "num_input_records": 3574, "num_output_records": 3061, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:23:39.126777 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286619126773, "job": 732, "event": "table_file_deletion", "file_number": 10224}
>2018-07-11 05:23:39.129321 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286619129319, "job": 732, "event": "table_file_deletion", "file_number": 10222}
>2018-07-11 05:23:42.612687 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216576 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:42.760230 7f29149e4700  1 mon.magna102@0(leader).log v19075 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19075)
>2018-07-11 05:23:49.652110 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216579 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:49.752474 7f29149e4700  1 mon.magna102@0(leader).log v19076 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19076)
>2018-07-11 05:23:54.652995 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216582 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:23:54.770024 7f29149e4700  1 mon.magna102@0(leader).log v19077 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19077)
>2018-07-11 05:24:00.292588 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:24:02.621387 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216585 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:02.762420 7f29149e4700  1 mon.magna102@0(leader).log v19078 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19078)
>2018-07-11 05:24:09.654226 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216588 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:09.754664 7f29149e4700  1 mon.magna102@0(leader).log v19079 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19079)
>2018-07-11 05:24:16.627378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216594 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:16.781657 7f29149e4700  1 mon.magna102@0(leader).log v19080 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19080)
>2018-07-11 05:24:26.631644 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216597 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:26.782502 7f29149e4700  1 mon.magna102@0(leader).log v19081 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19081)
>2018-07-11 05:24:34.634997 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216600 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:34.800016 7f29149e4700  1 mon.magna102@0(leader).log v19082 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19082)
>2018-07-11 05:24:39.674838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216603 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:39.792138 7f29149e4700  1 mon.magna102@0(leader).log v19083 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19083)
>2018-07-11 05:24:48.641020 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216609 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:48.809697 7f29149e4700  1 mon.magna102@0(leader).log v19084 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19084)
>2018-07-11 05:24:56.644445 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216612 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:24:56.810419 7f29149e4700  1 mon.magna102@0(leader).log v19085 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19085)
>2018-07-11 05:25:00.292775 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:25:02.646883 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216615 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:02.819338 7f29149e4700  1 mon.magna102@0(leader).log v19086 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19086)
>2018-07-11 05:25:10.650284 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216618 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:10.820175 7f29149e4700  1 mon.magna102@0(leader).log v19087 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19087)
>2018-07-11 05:25:16.652941 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216621 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:16.805334 7f29149e4700  1 mon.magna102@0(leader).log v19088 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19088)
>2018-07-11 05:25:22.655443 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216624 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:22.805912 7f29149e4700  1 mon.magna102@0(leader).log v19089 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19089)
>2018-07-11 05:25:28.657975 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216627 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:28.806521 7f29149e4700  1 mon.magna102@0(leader).log v19090 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19090)
>2018-07-11 05:25:34.696046 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216630 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:34.815479 7f29149e4700  1 mon.magna102@0(leader).log v19091 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19091)
>2018-07-11 05:25:39.696282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216633 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:39.799269 7f29149e4700  1 mon.magna102@0(leader).log v19092 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19092)
>2018-07-11 05:25:46.665927 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216636 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:46.816836 7f29149e4700  1 mon.magna102@0(leader).log v19093 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19093)
>2018-07-11 05:25:54.669253 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216639 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:25:54.817553 7f29149e4700  1 mon.magna102@0(leader).log v19094 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19094)
>2018-07-11 05:26:00.292955 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:26:00.671793 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216642 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:00.826488 7f29149e4700  1 mon.magna102@0(leader).log v19095 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19095)
>2018-07-11 05:26:09.710155 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216645 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:09.818857 7f29149e4700  1 mon.magna102@0(leader).log v19096 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19096)
>2018-07-11 05:26:16.678882 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216651 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:16.854215 7f29149e4700  1 mon.magna102@0(leader).log v19097 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19097)
>2018-07-11 05:26:24.682074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216654 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:24.855018 7f29149e4700  1 mon.magna102@0(leader).log v19098 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19098)
>2018-07-11 05:26:32.685612 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216657 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:32.855671 7f29149e4700  1 mon.magna102@0(leader).log v19099 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19099)
>2018-07-11 05:26:39.722608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216660 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:39.839679 7f29149e4700  1 mon.magna102@0(leader).log v19100 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19100)
>2018-07-11 05:26:44.739970 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216663 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:44.857006 7f29149e4700  1 mon.magna102@0(leader).log v19101 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19101)
>2018-07-11 05:26:49.740207 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216666 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:49.807700 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10226. Immutable memtables: 0.
>
>2018-07-11 05:26:49.807744 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 733] Syncing log #10223
>2018-07-11 05:26:49.824208 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.807734) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:26:49.824219 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 733] Flushing memtable with next log file: 10226
>
>2018-07-11 05:26:49.824246 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286809824238, "job": 733, "event": "flush_started", "num_memtables": 1, "num_entries": 1136, "num_deletes": 250, "memory_usage": 1949224}
>2018-07-11 05:26:49.824249 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 733] Level-0 flush table #10227: started
>2018-07-11 05:26:49.908299 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286809908284, "cf_name": "default", "job": 733, "event": "table_file_creation", "file_number": 10227, "file_size": 1269473, "table_properties": {"data_size": 1258877, "index_size": 5107, "filter_size": 4569, "raw_key_size": 9408, "raw_average_key_size": 21, "raw_value_size": 1249127, "raw_average_value_size": 2832, "num_data_blocks": 158, "num_entries": 441, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:26:49.908310 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 733] Level-0 flush table #10227: 1269473 bytes OK
>2018-07-11 05:26:49.941730 7f29149e4700  1 mon.magna102@0(leader).log v19102 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19102)
>2018-07-11 05:26:49.974912 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.941491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10227 started
>2018-07-11 05:26:49.974921 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.974875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10227: memtable #1 done
>2018-07-11 05:26:49.974925 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.974890) EVENT_LOG_v1 {"time_micros": 1531286809974884, "job": 733, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:26:49.974927 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.974902) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:26:49.974934 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 733] Try to delete WAL files size 1925930, prev total WAL file size 1944630, number of live WAL files 2.
>
>2018-07-11 05:26:49.975462 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:26:49.975511 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:49.975504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:26:49.975518 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 734] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:26:49.975524 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 733 Base level 0, inputs: [10227(1239KB)], [10225(13MB)]
>
>2018-07-11 05:26:49.975537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286809975527, "job": 734, "event": "compaction_started", "files_L0": [10227], "files_L1": [10225], "score": -1, "input_data_size": 15460136}
>2018-07-11 05:26:50.159035 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 734] Generated table #10228: 3083 keys, 14355643 bytes
>2018-07-11 05:26:50.159082 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810159055, "cf_name": "default", "job": 734, "event": "table_file_creation", "file_number": 10228, "file_size": 14355643, "table_properties": {"data_size": 14245150, "index_size": 62981, "filter_size": 46585, "raw_key_size": 61960, "raw_average_key_size": 20, "raw_value_size": 14157048, "raw_average_value_size": 4591, "num_data_blocks": 1846, "num_entries": 3083, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:26:50.200561 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 734] Compacted 1@0 + 1@1 files to L1 => 14355643 bytes
>2018-07-11 05:26:50.234077 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.234009) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.7 rd, 63.8 wr, level 1, files in(1, 1) out(1) MB in(1.2, 13.5) out(13.7), read-write-amplify(23.5) write-amplify(11.3) OK, records in: 3502, records dropped: 419
>
>2018-07-11 05:26:50.234086 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.234030) EVENT_LOG_v1 {"time_micros": 1531286810234021, "job": 734, "event": "compaction_finished", "compaction_time_micros": 225009, "output_level": 1, "num_output_files": 1, "total_output_size": 14355643, "num_input_records": 3502, "num_output_records": 3083, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:26:50.234419 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810234415, "job": 734, "event": "table_file_deletion", "file_number": 10227}
>2018-07-11 05:26:50.237113 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810237110, "job": 734, "event": "table_file_deletion", "file_number": 10225}
>2018-07-11 05:26:50.237474 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10229. Immutable memtables: 0.
>
>2018-07-11 05:26:50.237593 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 735] Syncing log #10226
>2018-07-11 05:26:50.251084 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.237584) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:26:50.251096 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 735] Flushing memtable with next log file: 10229
>
>2018-07-11 05:26:50.251122 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810251114, "job": 735, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21048}
>2018-07-11 05:26:50.251125 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 735] Level-0 flush table #10230: started
>2018-07-11 05:26:50.293245 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810293232, "cf_name": "default", "job": 735, "event": "table_file_creation", "file_number": 10230, "file_size": 19819, "table_properties": {"data_size": 18749, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18526, "raw_average_value_size": 2315, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:26:50.293255 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 735] Level-0 flush table #10230: 19819 bytes OK
>2018-07-11 05:26:50.343282 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.309759) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10230 started
>2018-07-11 05:26:50.343290 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.343248) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10230: memtable #1 done
>2018-07-11 05:26:50.343292 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.343262) EVENT_LOG_v1 {"time_micros": 1531286810343256, "job": 735, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:26:50.343308 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.343273) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:26:50.343313 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 735] Try to delete WAL files size 18700, prev total WAL file size 18700, number of live WAL files 2.
>
>2018-07-11 05:26:50.343413 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:26:50.343460 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.343453) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:26:50.343469 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 736] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:26:50.343475 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 735 Base level 0, inputs: [10230(19KB)], [10228(13MB)]
>
>2018-07-11 05:26:50.343488 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810343478, "job": 736, "event": "compaction_started", "files_L0": [10230], "files_L1": [10228], "score": -1, "input_data_size": 14375462}
>2018-07-11 05:26:50.520203 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 736] Generated table #10231: 3086 keys, 14374345 bytes
>2018-07-11 05:26:50.520233 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810520218, "cf_name": "default", "job": 736, "event": "table_file_creation", "file_number": 10231, "file_size": 14374345, "table_properties": {"data_size": 14263726, "index_size": 63050, "filter_size": 46642, "raw_key_size": 62021, "raw_average_key_size": 20, "raw_value_size": 14175534, "raw_average_value_size": 4593, "num_data_blocks": 1848, "num_entries": 3086, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:26:50.553452 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 736] Compacted 1@0 + 1@1 files to L1 => 14374345 bytes
>2018-07-11 05:26:50.586975 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.586908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.5 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.7) out(13.7), read-write-amplify(1450.6) write-amplify(725.3) OK, records in: 3091, records dropped: 5
>
>2018-07-11 05:26:50.586984 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:26:50.586930) EVENT_LOG_v1 {"time_micros": 1531286810586920, "job": 736, "event": "compaction_finished", "compaction_time_micros": 209948, "output_level": 1, "num_output_files": 1, "total_output_size": 14374345, "num_input_records": 3091, "num_output_records": 3086, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:26:50.587077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810587073, "job": 736, "event": "table_file_deletion", "file_number": 10230}
>2018-07-11 05:26:50.589827 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286810589824, "job": 736, "event": "table_file_deletion", "file_number": 10228}
>2018-07-11 05:26:54.774227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216669 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:26:55.829747 7f29149e4700  1 mon.magna102@0(leader).log v19103 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19103)
>2018-07-11 05:27:00.293171 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:27:02.698580 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216672 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:02.847326 7f29149e4700  1 mon.magna102@0(leader).log v19104 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19104)
>2018-07-11 05:27:09.774854 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216675 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:09.897871 7f29149e4700  1 mon.magna102@0(leader).log v19105 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19105)
>2018-07-11 05:27:14.775078 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216678 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:15.840144 7f29149e4700  1 mon.magna102@0(leader).log v19106 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19106)
>2018-07-11 05:27:19.775318 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216681 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:19.848983 7f29149e4700  1 mon.magna102@0(leader).log v19107 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19107)
>2018-07-11 05:27:28.709699 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216684 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:28.866549 7f29149e4700  1 mon.magna102@0(leader).log v19108 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19108)
>2018-07-11 05:27:34.775875 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216687 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:34.900608 7f29149e4700  1 mon.magna102@0(leader).log v19109 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19109)
>2018-07-11 05:27:40.714893 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216690 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:40.877458 7f29149e4700  1 mon.magna102@0(leader).log v19110 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19110)
>2018-07-11 05:27:49.776470 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216693 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:49.886491 7f29149e4700  1 mon.magna102@0(leader).log v19111 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19111)
>2018-07-11 05:27:54.776700 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216696 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:54.912156 7f29149e4700  1 mon.magna102@0(leader).log v19112 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19112)
>2018-07-11 05:27:59.776936 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216699 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:27:59.854294 7f29149e4700  1 mon.magna102@0(leader).log v19113 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19113)
>2018-07-11 05:28:00.293371 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:28:04.777213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216702 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:04.896608 7f29149e4700  1 mon.magna102@0(leader).log v19114 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19114)
>2018-07-11 05:28:10.727872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216705 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:10.880641 7f29149e4700  1 mon.magna102@0(leader).log v19115 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19115)
>2018-07-11 05:28:19.777831 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216708 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:19.864768 7f29149e4700  1 mon.magna102@0(leader).log v19116 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19116)
>2018-07-11 05:28:24.778087 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216711 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:24.898755 7f29149e4700  1 mon.magna102@0(leader).log v19117 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19117)
>2018-07-11 05:28:29.778338 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216714 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:29.874213 7f29149e4700  1 mon.magna102@0(leader).log v19118 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19118)
>2018-07-11 05:28:34.778587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216717 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:34.899870 7f29149e4700  1 mon.magna102@0(leader).log v19119 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19119)
>2018-07-11 05:28:40.740923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216720 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:40.893351 7f29149e4700  1 mon.magna102@0(leader).log v19120 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19120)
>2018-07-11 05:28:46.743592 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216723 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:46.893947 7f29149e4700  1 mon.magna102@0(leader).log v19121 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19121)
>2018-07-11 05:28:52.746117 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216726 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:52.902814 7f29149e4700  1 mon.magna102@0(leader).log v19122 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19122)
>2018-07-11 05:28:58.748622 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216729 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:28:58.903471 7f29149e4700  1 mon.magna102@0(leader).log v19123 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19123)
>2018-07-11 05:29:00.293550 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:29:04.751163 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216732 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:04.912400 7f29149e4700  1 mon.magna102@0(leader).log v19124 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19124)
>2018-07-11 05:29:10.753808 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216735 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:10.904662 7f29149e4700  1 mon.magna102@0(leader).log v19125 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19125)
>2018-07-11 05:29:16.756522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216738 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:16.930354 7f29149e4700  1 mon.magna102@0(leader).log v19126 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19126)
>2018-07-11 05:29:24.759991 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216741 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:24.931273 7f29149e4700  1 mon.magna102@0(leader).log v19127 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19127)
>2018-07-11 05:29:30.762377 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216744 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:30.931861 7f29149e4700  1 mon.magna102@0(leader).log v19128 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19128)
>2018-07-11 05:29:39.800075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216747 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:39.908797 7f29149e4700  1 mon.magna102@0(leader).log v19129 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19129)
>2018-07-11 05:29:44.800758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216750 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:44.917809 7f29149e4700  1 mon.magna102@0(leader).log v19130 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19130)
>2018-07-11 05:29:49.800990 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216753 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:49.901595 7f29149e4700  1 mon.magna102@0(leader).log v19131 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19131)
>2018-07-11 05:29:58.774421 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216759 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:29:58.927507 7f29149e4700  1 mon.magna102@0(leader).log v19132 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19132)
>2018-07-11 05:30:00.293732 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:30:00.911359 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10232. Immutable memtables: 0.
>
>2018-07-11 05:30:00.911403 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 737] Syncing log #10229
>2018-07-11 05:30:00.927918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:30:00.911394) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:30:00.927929 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 737] Flushing memtable with next log file: 10232
>
>2018-07-11 05:30:00.927956 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287000927947, "job": 737, "event": "flush_started", "num_memtables": 1, "num_entries": 1168, "num_deletes": 251, "memory_usage": 2024688}
>2018-07-11 05:30:00.927959 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 737] Level-0 flush table #10233: started
>2018-07-11 05:30:00.978467 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287000978452, "cf_name": "default", "job": 737, "event": "table_file_creation", "file_number": 10233, "file_size": 1951009, "table_properties": {"data_size": 1934923, "index_size": 8394, "filter_size": 6772, "raw_key_size": 11109, "raw_average_key_size": 20, "raw_value_size": 1921370, "raw_average_value_size": 3544, "num_data_blocks": 255, "num_entries": 542, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:30:00.978477 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 737] Level-0 flush table #10233: 1951009 bytes OK
>2018-07-11 05:30:01.028477 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:30:00.994971) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10233 started
>2018-07-11 05:30:01.028485 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.028441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10233: memtable #1 done
>2018-07-11 05:30:01.028488 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.028455) EVENT_LOG_v1 {"time_micros": 1531287001028449, "job": 737, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:30:01.028503 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.028467) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:30:01.028509 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 737] Try to delete WAL files size 2000377, prev total WAL file size 2000377, number of live WAL files 2.
>
>2018-07-11 05:30:01.029063 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:30:01.029125 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.029118) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:30:01.029132 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 738] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:30:01.029138 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 737 Base level 0, inputs: [10233(1905KB)], [10231(13MB)]
>
>2018-07-11 05:30:01.029153 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287001029143, "job": 738, "event": "compaction_started", "files_L0": [10233], "files_L1": [10231], "score": -1, "input_data_size": 16325354}
>2018-07-11 05:30:01.204157 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 738] Generated table #10234: 3118 keys, 14325245 bytes
>2018-07-11 05:30:01.204223 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287001204207, "cf_name": "default", "job": 738, "event": "table_file_creation", "file_number": 10234, "file_size": 14325245, "table_properties": {"data_size": 14214248, "index_size": 63430, "filter_size": 46640, "raw_key_size": 62883, "raw_average_key_size": 20, "raw_value_size": 14125272, "raw_average_value_size": 4530, "num_data_blocks": 1851, "num_entries": 3118, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:30:01.237228 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 738] Compacted 1@0 + 1@1 files to L1 => 14325245 bytes
>2018-07-11 05:30:01.279210 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.279137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 78.5 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(1.9, 13.7) out(13.7), read-write-amplify(15.7) write-amplify(7.3) OK, records in: 3628, records dropped: 510
>
>2018-07-11 05:30:01.279219 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:30:01.279158) EVENT_LOG_v1 {"time_micros": 1531287001279149, "job": 738, "event": "compaction_finished", "compaction_time_micros": 208054, "output_level": 1, "num_output_files": 1, "total_output_size": 14325245, "num_input_records": 3628, "num_output_records": 3118, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:30:01.279682 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287001279677, "job": 738, "event": "table_file_deletion", "file_number": 10233}
>2018-07-11 05:30:01.282481 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287001282478, "job": 738, "event": "table_file_deletion", "file_number": 10231}
>2018-07-11 05:30:04.777263 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216762 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:04.930460 7f29149e4700  1 mon.magna102@0(leader).log v19133 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19133)
>2018-07-11 05:30:14.781590 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216765 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:14.939620 7f29149e4700  1 mon.magna102@0(leader).log v19134 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19134)
>2018-07-11 05:30:19.822881 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216768 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:19.916342 7f29149e4700  1 mon.magna102@0(leader).log v19135 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19135)
>2018-07-11 05:30:24.823106 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216771 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:24.942033 7f29149e4700  1 mon.magna102@0(leader).log v19136 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19136)
>2018-07-11 05:30:29.823345 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216774 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:29.925648 7f29149e4700  1 mon.magna102@0(leader).log v19137 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19137)
>2018-07-11 05:30:36.791344 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216777 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:36.960233 7f29149e4700  1 mon.magna102@0(leader).log v19138 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19138)
>2018-07-11 05:30:44.794657 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216780 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:44.952646 7f29149e4700  1 mon.magna102@0(leader).log v19139 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19139)
>2018-07-11 05:30:50.797258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216783 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:50.961544 7f29149e4700  1 mon.magna102@0(leader).log v19140 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19140)
>2018-07-11 05:30:59.836894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216786 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:30:59.962264 7f29149e4700  1 mon.magna102@0(leader).log v19141 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19141)
>2018-07-11 05:31:00.293905 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:31:04.845895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216789 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:04.979602 7f29149e4700  1 mon.magna102@0(leader).log v19142 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19142)
>2018-07-11 05:31:09.846108 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216792 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:09.971807 7f29149e4700  1 mon.magna102@0(leader).log v19143 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19143)
>2018-07-11 05:31:18.809464 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216798 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:18.957283 7f29149e4700  1 mon.magna102@0(leader).log v19144 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19144)
>2018-07-11 05:31:26.812928 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216801 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:26.966342 7f29149e4700  1 mon.magna102@0(leader).log v19145 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19145)
>2018-07-11 05:31:32.815530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216804 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:32.966869 7f29149e4700  1 mon.magna102@0(leader).log v19146 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19146)
>2018-07-11 05:31:40.818998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216807 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:40.967760 7f29149e4700  1 mon.magna102@0(leader).log v19147 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19147)
>2018-07-11 05:31:49.859832 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216810 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:49.960241 7f29149e4700  1 mon.magna102@0(leader).log v19148 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19148)
>2018-07-11 05:31:54.860557 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216813 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:54.977569 7f29149e4700  1 mon.magna102@0(leader).log v19149 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19149)
>2018-07-11 05:31:59.860789 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216816 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:31:59.969686 7f29149e4700  1 mon.magna102@0(leader).log v19150 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19150)
>2018-07-11 05:32:00.294094 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:32:04.869992 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216819 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:04.987021 7f29149e4700  1 mon.magna102@0(leader).log v19151 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19151)
>2018-07-11 05:32:12.833000 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216822 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:13.005641 7f29149e4700  1 mon.magna102@0(leader).log v19152 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19152)
>2018-07-11 05:32:19.872493 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216825 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:19.997895 7f29149e4700  1 mon.magna102@0(leader).log v19153 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19153)
>2018-07-11 05:32:24.881629 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216828 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:25.015318 7f29149e4700  1 mon.magna102@0(leader).log v19154 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19154)
>2018-07-11 05:32:30.840701 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216831 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:30.990899 7f29149e4700  1 mon.magna102@0(leader).log v19155 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19155)
>2018-07-11 05:32:39.882859 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216834 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:39.974889 7f29149e4700  1 mon.magna102@0(leader).log v19156 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19156)
>2018-07-11 05:32:44.883091 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216837 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:45.000571 7f29149e4700  1 mon.magna102@0(leader).log v19157 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19157)
>2018-07-11 05:32:50.849431 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216840 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:51.001155 7f29149e4700  1 mon.magna102@0(leader).log v19158 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19158)
>2018-07-11 05:32:59.884865 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216843 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:32:59.993620 7f29149e4700  1 mon.magna102@0(leader).log v19159 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19159)
>2018-07-11 05:33:00.294270 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:33:04.885577 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216846 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:05.002569 7f29149e4700  1 mon.magna102@0(leader).log v19160 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19160)
>2018-07-11 05:33:09.885806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216849 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:09.995966 7f29149e4700  1 mon.magna102@0(leader).log v19161 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19161)
>2018-07-11 05:33:14.896258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216852 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:15.013290 7f29149e4700  1 mon.magna102@0(leader).log v19162 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19162)
>2018-07-11 05:33:22.863525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216855 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:23.014119 7f29149e4700  1 mon.magna102@0(leader).log v19163 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19163)
>2018-07-11 05:33:29.897621 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216858 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:30.023026 7f29149e4700  1 mon.magna102@0(leader).log v19164 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19164)
>2018-07-11 05:33:34.906765 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216861 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:35.023807 7f29149e4700  1 mon.magna102@0(leader).log v19165 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19165)
>2018-07-11 05:33:39.906971 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216864 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:40.007571 7f29149e4700  1 mon.magna102@0(leader).log v19166 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19166)
>2018-07-11 05:33:49.916376 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216867 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:50.033464 7f29149e4700  1 mon.magna102@0(leader).log v19167 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19167)
>2018-07-11 05:33:54.916603 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216870 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:33:55.050659 7f29149e4700  1 mon.magna102@0(leader).log v19168 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19168)
>2018-07-11 05:33:59.916819 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216873 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:00.042907 7f29149e4700  1 mon.magna102@0(leader).log v19169 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19169)
>2018-07-11 05:34:00.294446 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:34:04.917056 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216876 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:05.036482 7f29149e4700  1 mon.magna102@0(leader).log v19170 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19170)
>2018-07-11 05:34:10.884261 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216879 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:11.037177 7f29149e4700  1 mon.magna102@0(leader).log v19171 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19171)
>2018-07-11 05:34:16.886905 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216882 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:17.037767 7f29149e4700  1 mon.magna102@0(leader).log v19172 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19172)
>2018-07-11 05:34:24.921519 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216885 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:25.038519 7f29149e4700  1 mon.magna102@0(leader).log v19173 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19173)
>2018-07-11 05:34:29.921741 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216888 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:30.022315 7f29149e4700  1 mon.magna102@0(leader).log v19174 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19174)
>2018-07-11 05:34:36.895431 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216891 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:37.048135 7f29149e4700  1 mon.magna102@0(leader).log v19175 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19175)
>2018-07-11 05:34:42.898140 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216894 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:43.048591 7f29149e4700  1 mon.magna102@0(leader).log v19176 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19176)
>2018-07-11 05:34:49.940615 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216897 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:50.032669 7f29149e4700  1 mon.magna102@0(leader).log v19177 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19177)
>2018-07-11 05:34:58.904948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216903 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:34:59.059814 7f29149e4700  1 mon.magna102@0(leader).log v19178 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19178)
>2018-07-11 05:35:00.294626 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:35:06.908484 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216906 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:07.068847 7f29149e4700  1 mon.magna102@0(leader).log v19179 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19179)
>2018-07-11 05:35:10.011111 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10235. Immutable memtables: 0.
>
>2018-07-11 05:35:10.011154 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 739] Syncing log #10232
>2018-07-11 05:35:10.027579 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.011144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:35:10.027590 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 739] Flushing memtable with next log file: 10235
>
>2018-07-11 05:35:10.027617 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310027608, "job": 739, "event": "flush_started", "num_memtables": 1, "num_entries": 1709, "num_deletes": 250, "memory_usage": 3215352}
>2018-07-11 05:35:10.027619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 739] Level-0 flush table #10236: started
>2018-07-11 05:35:10.086462 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310086448, "cf_name": "default", "job": 739, "event": "table_file_creation", "file_number": 10236, "file_size": 2090575, "table_properties": {"data_size": 2074195, "index_size": 8357, "filter_size": 7103, "raw_key_size": 11808, "raw_average_key_size": 21, "raw_value_size": 2060424, "raw_average_value_size": 3679, "num_data_blocks": 258, "num_entries": 560, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:35:10.086473 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 739] Level-0 flush table #10236: 2090575 bytes OK
>2018-07-11 05:35:10.136473 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.102980) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10236 started
>2018-07-11 05:35:10.136481 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.136436) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10236: memtable #1 done
>2018-07-11 05:35:10.136483 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.136450) EVENT_LOG_v1 {"time_micros": 1531287310136444, "job": 739, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:35:10.136499 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.136462) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:35:10.136505 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 739] Try to delete WAL files size 3181753, prev total WAL file size 3181753, number of live WAL files 2.
>
>2018-07-11 05:35:10.137302 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:35:10.137484 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 05:35:10.137491 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 49337.0 total, 691.3 interval
>Cumulative writes: 69K writes, 331K keys, 69K commit groups, 1.0 writes per commit group, ingest: 0.55 GB, 0.01 MB/s
>Cumulative WAL: 69K writes, 69K syncs, 1.00 writes per sync, written: 0.55 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 906 writes, 4021 keys, 906 commit groups, 1.0 writes per commit group, ingest: 6.80 MB, 0.01 MB/s
>Interval WAL: 906 writes, 906 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.99 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        31       370    0.083       0      0
>  L1      1/1   13.66 MB   0.0      5.8     0.5      5.3       5.3     -0.0       0.0  10.9     72.2     66.1        82       368    0.222   1300K   153K
> Sum      2/2   15.66 MB   0.0      5.8     0.5      5.3       5.8      0.5       0.0  11.9     52.5     52.5       112       738    0.152   1300K   153K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.7     50.5     50.7         1         8    0.147     13K   1447
>Uptime(secs): 49337.0 total, 49337.0 interval
>Flush(GB): cumulative 0.486, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.76 GB write, 0.12 MB/s write, 5.76 GB read, 0.12 MB/s read, 112.4 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.99 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        31       370    0.083       0      0
>  L1      1/1   13.66 MB   0.0      5.8     0.5      5.3       5.3     -0.0       0.0  10.9     72.2     66.1        82       368    0.222   1300K   153K
> Sum      2/2   15.66 MB   0.0      5.8     0.5      5.3       5.8      0.5       0.0  11.9     52.5     52.5       112       738    0.152   1300K   153K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 49337.0 total, 49337.0 interval
>Flush(GB): cumulative 0.486, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.76 GB write, 0.12 MB/s write, 5.76 GB read, 0.12 MB/s read, 112.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 05:35:10.137516 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.137510) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:35:10.137521 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 740] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:35:10.137528 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 739 Base level 0, inputs: [10236(2041KB)], [10234(13MB)]
>
>2018-07-11 05:35:10.137549 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310137532, "job": 740, "event": "compaction_started", "files_L0": [10236], "files_L1": [10234], "score": -1, "input_data_size": 16415820}
>2018-07-11 05:35:10.328815 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 740] Generated table #10237: 3318 keys, 15714213 bytes
>2018-07-11 05:35:10.328860 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310328833, "cf_name": "default", "job": 740, "event": "table_file_creation", "file_number": 10237, "file_size": 15714213, "table_properties": {"data_size": 15593944, "index_size": 68503, "filter_size": 50839, "raw_key_size": 66738, "raw_average_key_size": 20, "raw_value_size": 15498637, "raw_average_value_size": 4671, "num_data_blocks": 2007, "num_entries": 3318, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:35:10.362033 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 740] Compacted 1@0 + 1@1 files to L1 => 15714213 bytes
>2018-07-11 05:35:10.395500 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.395417) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.1 rd, 70.0 wr, level 1, files in(1, 1) out(1) MB in(2.0, 13.7) out(15.0), read-write-amplify(15.4) write-amplify(7.5) OK, records in: 3678, records dropped: 360
>
>2018-07-11 05:35:10.395509 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:35:10.395440) EVENT_LOG_v1 {"time_micros": 1531287310395430, "job": 740, "event": "compaction_finished", "compaction_time_micros": 224467, "output_level": 1, "num_output_files": 1, "total_output_size": 15714213, "num_input_records": 3678, "num_output_records": 3318, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:35:10.395994 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310395990, "job": 740, "event": "table_file_deletion", "file_number": 10236}
>2018-07-11 05:35:10.398737 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310398734, "job": 740, "event": "table_file_deletion", "file_number": 10234}
>2018-07-11 05:35:10.398854 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:35:14.977601 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216909 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:15.096710 7f29149e4700  1 mon.magna102@0(leader).log v19180 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19180)
>2018-07-11 05:35:19.977878 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216912 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:20.047164 7f29149e4700  1 mon.magna102@0(leader).log v19181 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19181)
>2018-07-11 05:35:24.978103 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216915 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:25.097915 7f29149e4700  1 mon.magna102@0(leader).log v19182 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19182)
>2018-07-11 05:35:30.918846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216918 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:31.073508 7f29149e4700  1 mon.magna102@0(leader).log v19183 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19183)
>2018-07-11 05:35:38.922369 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216921 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:39.066006 7f29149e4700  1 mon.magna102@0(leader).log v19184 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19184)
>2018-07-11 05:35:44.924958 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216924 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:45.101162 7f29149e4700  1 mon.magna102@0(leader).log v19185 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19185)
>2018-07-11 05:35:49.979095 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216927 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:50.093292 7f29149e4700  1 mon.magna102@0(leader).log v19186 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19186)
>2018-07-11 05:35:56.930338 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216930 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:35:57.102385 7f29149e4700  1 mon.magna102@0(leader).log v19187 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19187)
>2018-07-11 05:36:00.294797 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 05:36:02.932900 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216933 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:03.086306 7f29149e4700  1 mon.magna102@0(leader).log v19188 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19188)
>2018-07-11 05:36:09.979872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216936 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:10.070253 7f29149e4700  1 mon.magna102@0(leader).log v19189 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19189)
>2018-07-11 05:36:14.980103 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216939 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:15.095897 7f29149e4700  1 mon.magna102@0(leader).log v19190 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19190)
>2018-07-11 05:36:20.940401 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216942 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:21.088138 7f29149e4700  1 mon.magna102@0(leader).log v19191 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19191)
>2018-07-11 05:36:21.155254 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10238. Immutable memtables: 0.
>
>2018-07-11 05:36:21.155297 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 741] Syncing log #10235
>2018-07-11 05:36:21.171792 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.155287) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:36:21.171803 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 741] Flushing memtable with next log file: 10238
>
>2018-07-11 05:36:21.171830 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381171822, "job": 741, "event": "flush_started", "num_memtables": 1, "num_entries": 604, "num_deletes": 251, "memory_usage": 785616}
>2018-07-11 05:36:21.171833 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 741] Level-0 flush table #10239: started
>2018-07-11 05:36:21.213997 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381213983, "cf_name": "default", "job": 741, "event": "table_file_creation", "file_number": 10239, "file_size": 754652, "table_properties": {"data_size": 747518, "index_size": 3214, "filter_size": 3001, "raw_key_size": 7491, "raw_average_key_size": 20, "raw_value_size": 740178, "raw_average_value_size": 2011, "num_data_blocks": 98, "num_entries": 368, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:36:21.214008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 741] Level-0 flush table #10239: 754652 bytes OK
>2018-07-11 05:36:21.272364 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.230518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10239 started
>2018-07-11 05:36:21.272373 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.272329) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10239: memtable #1 done
>2018-07-11 05:36:21.272387 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.272343) EVENT_LOG_v1 {"time_micros": 1531287381272337, "job": 741, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:36:21.272390 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.272355) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:36:21.272396 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 741] Try to delete WAL files size 771893, prev total WAL file size 771893, number of live WAL files 2.
>
>2018-07-11 05:36:21.272688 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:36:21.272740 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.272732) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:36:21.272749 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 742] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:36:21.272756 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 741 Base level 0, inputs: [10239(736KB)], [10237(14MB)]
>
>2018-07-11 05:36:21.272769 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381272759, "job": 742, "event": "compaction_started", "files_L0": [10239], "files_L1": [10237], "score": -1, "input_data_size": 16468865}
>2018-07-11 05:36:21.439720 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 742] Generated table #10240: 3176 keys, 14468118 bytes
>2018-07-11 05:36:21.439768 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381439740, "cf_name": "default", "job": 742, "event": "table_file_creation", "file_number": 10240, "file_size": 14468118, "table_properties": {"data_size": 14356589, "index_size": 63545, "filter_size": 47057, "raw_key_size": 63982, "raw_average_key_size": 20, "raw_value_size": 14266716, "raw_average_value_size": 4492, "num_data_blocks": 1853, "num_entries": 3176, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:36:21.472937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 742] Compacted 1@0 + 1@1 files to L1 => 14468118 bytes
>2018-07-11 05:36:21.506450 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.506383) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 82.3 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.0) out(13.8), read-write-amplify(41.0) write-amplify(19.2) OK, records in: 3686, records dropped: 510
>
>2018-07-11 05:36:21.506459 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:36:21.506405) EVENT_LOG_v1 {"time_micros": 1531287381506395, "job": 742, "event": "compaction_finished", "compaction_time_micros": 200153, "output_level": 1, "num_output_files": 1, "total_output_size": 14468118, "num_input_records": 3686, "num_output_records": 3176, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:36:21.506694 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381506690, "job": 742, "event": "table_file_deletion", "file_number": 10239}
>2018-07-11 05:36:21.509649 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287381509646, "job": 742, "event": "table_file_deletion", "file_number": 10237}
>2018-07-11 05:36:26.943382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216945 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:27.092010 7f29149e4700  1 mon.magna102@0(leader).log v19192 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19192)
>2018-07-11 05:36:32.946065 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216948 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:33.109334 7f29149e4700  1 mon.magna102@0(leader).log v19193 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19193)
>2018-07-11 05:36:39.984440 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216951 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:40.093348 7f29149e4700  1 mon.magna102@0(leader).log v19194 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19194)
>2018-07-11 05:36:48.952755 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216954 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:49.102549 7f29149e4700  1 mon.magna102@0(leader).log v19195 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19195)
>2018-07-11 05:36:54.994479 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216957 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:36:55.119869 7f29149e4700  1 mon.magna102@0(leader).log v19196 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19196)
>2018-07-11 05:36:59.994696 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216960 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:00.120317 7f29149e4700  1 mon.magna102@0(leader).log v19197 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19197)
>2018-07-11 05:37:00.294977 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:37:09.995831 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216963 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:10.129537 7f29149e4700  1 mon.magna102@0(leader).log v19198 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19198)
>2018-07-11 05:37:14.996684 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216966 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:15.130387 7f29149e4700  1 mon.magna102@0(leader).log v19199 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19199)
>2018-07-11 05:37:19.996923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216969 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:20.114159 7f29149e4700  1 mon.magna102@0(leader).log v19200 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19200)
>2018-07-11 05:37:30.007646 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216972 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:30.099654 7f29149e4700  1 mon.magna102@0(leader).log v19201 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19201)
>2018-07-11 05:37:35.008360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216975 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:35.125382 7f29149e4700  1 mon.magna102@0(leader).log v19202 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19202)
>2018-07-11 05:37:40.008555 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216978 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:40.109167 7f29149e4700  1 mon.magna102@0(leader).log v19203 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19203)
>2018-07-11 05:37:45.009506 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216981 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:45.126521 7f29149e4700  1 mon.magna102@0(leader).log v19204 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19204)
>2018-07-11 05:37:50.009725 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216984 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:37:50.118627 7f29149e4700  1 mon.magna102@0(leader).log v19205 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19205)
>2018-07-11 05:38:00.019104 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216987 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:00.111196 7f29149e4700  1 mon.magna102@0(leader).log v19206 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19206)
>2018-07-11 05:38:00.295154 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:38:05.019764 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216990 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:05.136860 7f29149e4700  1 mon.magna102@0(leader).log v19207 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19207)
>2018-07-11 05:38:10.019952 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216993 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:10.120561 7f29149e4700  1 mon.magna102@0(leader).log v19208 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19208)
>2018-07-11 05:38:15.029272 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216996 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:15.146309 7f29149e4700  1 mon.magna102@0(leader).log v19209 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19209)
>2018-07-11 05:38:22.993503 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24216999 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:23.165007 7f29149e4700  1 mon.magna102@0(leader).log v19210 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19210)
>2018-07-11 05:38:30.029777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217002 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:30.157342 7f29149e4700  1 mon.magna102@0(leader).log v19211 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19211)
>2018-07-11 05:38:35.032628 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217005 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:35.166336 7f29149e4700  1 mon.magna102@0(leader).log v19212 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19212)
>2018-07-11 05:38:40.032836 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217008 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:40.166784 7f29149e4700  1 mon.magna102@0(leader).log v19213 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19213)
>2018-07-11 05:38:47.003973 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217011 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:47.150886 7f29149e4700  1 mon.magna102@0(leader).log v19214 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19214)
>2018-07-11 05:38:55.034578 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217014 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:38:55.151642 7f29149e4700  1 mon.magna102@0(leader).log v19215 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19215)
>2018-07-11 05:39:00.034794 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217017 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:00.152093 7f29149e4700  1 mon.magna102@0(leader).log v19216 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19216)
>2018-07-11 05:39:00.295355 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:39:05.052394 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217020 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:05.169413 7f29149e4700  1 mon.magna102@0(leader).log v19217 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19217)
>2018-07-11 05:39:10.052614 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217023 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:10.144833 7f29149e4700  1 mon.magna102@0(leader).log v19218 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19218)
>2018-07-11 05:39:20.054943 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217026 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:20.155348 7f29149e4700  1 mon.magna102@0(leader).log v19219 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19219)
>2018-07-11 05:39:25.055642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217029 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:25.172673 7f29149e4700  1 mon.magna102@0(leader).log v19220 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19220)
>2018-07-11 05:39:30.055852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217032 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:30.156445 7f29149e4700  1 mon.magna102@0(leader).log v19221 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19221)
>2018-07-11 05:39:35.065081 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217035 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:35.165408 7f29149e4700  1 mon.magna102@0(leader).log v19222 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19222)
>2018-07-11 05:39:36.232296 7f29149e4700  1 mon.magna102@0(leader).log v19223 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19223)
>2018-07-11 05:39:39.207979 7f29149e4700  1 mon.magna102@0(leader).log v19224 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19224)
>2018-07-11 05:39:40.065269 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217038 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:40.283216 7f29149e4700  1 mon.magna102@0(leader).log v19225 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19225)
>2018-07-11 05:39:47.029727 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217041 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:47.184080 7f29149e4700  1 mon.magna102@0(leader).log v19226 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19226)
>2018-07-11 05:39:55.033054 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217044 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:39:55.184772 7f29149e4700  1 mon.magna102@0(leader).log v19227 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19227)
>2018-07-11 05:39:59.201877 7f29149e4700  1 mon.magna102@0(leader).log v19228 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19228)
>2018-07-11 05:40:00.295538 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:40:01.035588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217047 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:01.210572 7f29149e4700  1 mon.magna102@0(leader).log v19229 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19229)
>2018-07-11 05:40:06.179135 7f29149e4700  1 mon.magna102@0(leader).log v19230 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19230)
>2018-07-11 05:40:10.079056 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217050 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:10.254668 7f29149e4700  1 mon.magna102@0(leader).log v19231 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19231)
>2018-07-11 05:40:15.079889 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217053 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:15.196987 7f29149e4700  1 mon.magna102@0(leader).log v19232 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19232)
>2018-07-11 05:40:20.080108 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217056 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:20.189098 7f29149e4700  1 mon.magna102@0(leader).log v19233 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19233)
>2018-07-11 05:40:25.089402 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217059 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:25.206419 7f29149e4700  1 mon.magna102@0(leader).log v19234 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19234)
>2018-07-11 05:40:30.089594 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217062 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:30.181888 7f29149e4700  1 mon.magna102@0(leader).log v19235 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19235)
>2018-07-11 05:40:40.091013 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217065 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:40.191474 7f29149e4700  1 mon.magna102@0(leader).log v19236 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19236)
>2018-07-11 05:40:50.091901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217068 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:50.209159 7f29149e4700  1 mon.magna102@0(leader).log v19237 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19237)
>2018-07-11 05:40:55.092721 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217071 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:40:55.209673 7f29149e4700  1 mon.magna102@0(leader).log v19238 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19238)
>2018-07-11 05:41:00.092918 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217074 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:00.219828 7f29149e4700  1 mon.magna102@0(leader).log v19239 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19239)
>2018-07-11 05:41:00.295713 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:41:05.103464 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217077 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:05.237191 7f29149e4700  1 mon.magna102@0(leader).log v19240 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19240)
>2018-07-11 05:41:10.103681 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217080 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:10.229289 7f29149e4700  1 mon.magna102@0(leader).log v19241 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19241)
>2018-07-11 05:41:15.104592 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217083 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:15.238274 7f29149e4700  1 mon.magna102@0(leader).log v19242 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19242)
>2018-07-11 05:41:20.104790 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217086 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:20.238741 7f29149e4700  1 mon.magna102@0(leader).log v19243 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19243)
>2018-07-11 05:41:27.072901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217089 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:27.222802 7f29149e4700  1 mon.magna102@0(leader).log v19244 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19244)
>2018-07-11 05:41:33.075520 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217092 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:33.223535 7f29149e4700  1 mon.magna102@0(leader).log v19245 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19245)
>2018-07-11 05:41:40.114528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217095 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:40.207437 7f29149e4700  1 mon.magna102@0(leader).log v19246 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19246)
>2018-07-11 05:41:45.116054 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217098 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:45.233120 7f29149e4700  1 mon.magna102@0(leader).log v19247 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19247)
>2018-07-11 05:41:53.084248 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217101 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:41:53.235062 7f29149e4700  1 mon.magna102@0(leader).log v19248 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19248)
>2018-07-11 05:42:00.126846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217104 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:00.235847 7f29149e4700  1 mon.magna102@0(leader).log v19249 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19249)
>2018-07-11 05:42:00.295856 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:42:05.127082 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217107 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:05.244837 7f29149e4700  1 mon.magna102@0(leader).log v19250 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19250)
>2018-07-11 05:42:13.092991 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217110 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:13.245580 7f29149e4700  1 mon.magna102@0(leader).log v19251 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19251)
>2018-07-11 05:42:19.095442 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217113 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:19.246135 7f29149e4700  1 mon.magna102@0(leader).log v19252 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19252)
>2018-07-11 05:42:25.138087 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217116 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:25.255120 7f29149e4700  1 mon.magna102@0(leader).log v19253 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19253)
>2018-07-11 05:42:30.138271 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217119 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:30.247158 7f29149e4700  1 mon.magna102@0(leader).log v19254 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19254)
>2018-07-11 05:42:35.256333 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10241. Immutable memtables: 0.
>
>2018-07-11 05:42:35.256376 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 743] Syncing log #10238
>2018-07-11 05:42:35.272884 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.256367) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:42:35.272898 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 743] Flushing memtable with next log file: 10241
>
>2018-07-11 05:42:35.272912 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755272904, "job": 743, "event": "flush_started", "num_memtables": 1, "num_entries": 2069, "num_deletes": 251, "memory_usage": 3983544}
>2018-07-11 05:42:35.272915 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 743] Level-0 flush table #10242: started
>2018-07-11 05:42:35.340135 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755340107, "cf_name": "default", "job": 743, "event": "table_file_creation", "file_number": 10242, "file_size": 3844457, "table_properties": {"data_size": 3813913, "index_size": 16857, "filter_size": 12761, "raw_key_size": 16944, "raw_average_key_size": 20, "raw_value_size": 3790405, "raw_average_value_size": 4605, "num_data_blocks": 505, "num_entries": 823, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:42:35.340148 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 743] Level-0 flush table #10242: 3844457 bytes OK
>2018-07-11 05:42:35.390177 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.356702) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10242 started
>2018-07-11 05:42:35.390192 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.390137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10242: memtable #1 done
>2018-07-11 05:42:35.390194 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.390152) EVENT_LOG_v1 {"time_micros": 1531287755390146, "job": 743, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:42:35.390209 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.390166) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:42:35.390216 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 743] Try to delete WAL files size 3942356, prev total WAL file size 3942356, number of live WAL files 2.
>
>2018-07-11 05:42:35.391158 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:42:35.391234 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.391227) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:42:35.391244 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 744] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:42:35.391250 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 743 Base level 0, inputs: [10242(3754KB)], [10240(13MB)]
>
>2018-07-11 05:42:35.391264 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755391253, "job": 744, "event": "compaction_started", "files_L0": [10242], "files_L1": [10240], "score": -1, "input_data_size": 18312575}
>2018-07-11 05:42:35.582562 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 744] Generated table #10243: 3489 keys, 16310853 bytes
>2018-07-11 05:42:35.582603 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755582577, "cf_name": "default", "job": 744, "event": "table_file_creation", "file_number": 10243, "file_size": 16310853, "table_properties": {"data_size": 16184464, "index_size": 72422, "filter_size": 53040, "raw_key_size": 70679, "raw_average_key_size": 20, "raw_value_size": 16083835, "raw_average_value_size": 4609, "num_data_blocks": 2106, "num_entries": 3489, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:42:35.615745 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 744] Compacted 1@0 + 1@1 files to L1 => 16310853 bytes
>2018-07-11 05:42:35.649285 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.649218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.6 rd, 72.7 wr, level 1, files in(1, 1) out(1) MB in(3.7, 13.8) out(15.6), read-write-amplify(9.0) write-amplify(4.2) OK, records in: 3999, records dropped: 510
>
>2018-07-11 05:42:35.649293 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:42:35.649239) EVENT_LOG_v1 {"time_micros": 1531287755649229, "job": 744, "event": "compaction_finished", "compaction_time_micros": 224465, "output_level": 1, "num_output_files": 1, "total_output_size": 16310853, "num_input_records": 3999, "num_output_records": 3489, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:42:35.650093 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755650089, "job": 744, "event": "table_file_deletion", "file_number": 10242}
>2018-07-11 05:42:35.652923 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287755652920, "job": 744, "event": "table_file_deletion", "file_number": 10240}
>2018-07-11 05:42:39.104379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217125 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:39.276410 7f29149e4700  1 mon.magna102@0(leader).log v19255 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19255)
>2018-07-11 05:42:51.109523 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217128 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:51.285757 7f29149e4700  1 mon.magna102@0(leader).log v19256 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19256)
>2018-07-11 05:42:59.113132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217131 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:42:59.286478 7f29149e4700  1 mon.magna102@0(leader).log v19257 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19257)
>2018-07-11 05:43:00.296026 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 05:43:05.161677 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217134 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:05.287100 7f29149e4700  1 mon.magna102@0(leader).log v19258 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19258)
>2018-07-11 05:43:10.161887 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217137 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:10.270974 7f29149e4700  1 mon.magna102@0(leader).log v19259 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19259)
>2018-07-11 05:43:17.120855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217140 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:17.280079 7f29149e4700  1 mon.magna102@0(leader).log v19260 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19260)
>2018-07-11 05:43:23.123528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217143 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:23.272344 7f29149e4700  1 mon.magna102@0(leader).log v19261 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19261)
>2018-07-11 05:43:30.162638 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217146 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:30.231376 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10244. Immutable memtables: 0.
>
>2018-07-11 05:43:30.231421 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 745] Syncing log #10241
>2018-07-11 05:43:30.248021 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.231411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:43:30.248033 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 745] Flushing memtable with next log file: 10244
>
>2018-07-11 05:43:30.248046 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810248038, "job": 745, "event": "flush_started", "num_memtables": 1, "num_entries": 500, "num_deletes": 250, "memory_usage": 550448}
>2018-07-11 05:43:30.248048 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 745] Level-0 flush table #10245: started
>2018-07-11 05:43:30.291553 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810291539, "cf_name": "default", "job": 745, "event": "table_file_creation", "file_number": 10245, "file_size": 382728, "table_properties": {"data_size": 378408, "index_size": 1575, "filter_size": 1826, "raw_key_size": 6828, "raw_average_key_size": 21, "raw_value_size": 373011, "raw_average_value_size": 1191, "num_data_blocks": 49, "num_entries": 313, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:43:30.291564 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 745] Level-0 flush table #10245: 382728 bytes OK
>2018-07-11 05:43:30.375066 7f29149e4700  1 mon.magna102@0(leader).log v19262 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19262)
>2018-07-11 05:43:30.391552 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.341466) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10245 started
>2018-07-11 05:43:30.391561 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.391515) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10245: memtable #1 done
>2018-07-11 05:43:30.391565 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.391530) EVENT_LOG_v1 {"time_micros": 1531287810391524, "job": 745, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:43:30.391567 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.391542) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:43:30.391573 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 745] Try to delete WAL files size 539082, prev total WAL file size 556806, number of live WAL files 2.
>
>2018-07-11 05:43:30.391780 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:43:30.391866 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.391858) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:43:30.391878 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 746] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:43:30.391885 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 745 Base level 0, inputs: [10245(373KB)], [10243(15MB)]
>
>2018-07-11 05:43:30.391899 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810391889, "job": 746, "event": "compaction_started", "files_L0": [10245], "files_L1": [10243], "score": -1, "input_data_size": 16693581}
>2018-07-11 05:43:30.567360 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 746] Generated table #10246: 3315 keys, 15124967 bytes
>2018-07-11 05:43:30.567390 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810567376, "cf_name": "default", "job": 746, "event": "table_file_creation", "file_number": 10246, "file_size": 15124967, "table_properties": {"data_size": 15008833, "index_size": 66074, "filter_size": 49133, "raw_key_size": 66760, "raw_average_key_size": 20, "raw_value_size": 14915213, "raw_average_value_size": 4499, "num_data_blocks": 1926, "num_entries": 3315, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:43:30.600501 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 746] Compacted 1@0 + 1@1 files to L1 => 15124967 bytes
>2018-07-11 05:43:30.634032 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.633962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.0 rd, 72.5 wr, level 1, files in(1, 1) out(1) MB in(0.4, 15.6) out(14.4), read-write-amplify(83.1) write-amplify(39.5) OK, records in: 3802, records dropped: 487
>
>2018-07-11 05:43:30.634041 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.633983) EVENT_LOG_v1 {"time_micros": 1531287810633974, "job": 746, "event": "compaction_finished", "compaction_time_micros": 208588, "output_level": 1, "num_output_files": 1, "total_output_size": 15124967, "num_input_records": 3802, "num_output_records": 3315, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:43:30.634207 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810634203, "job": 746, "event": "table_file_deletion", "file_number": 10245}
>2018-07-11 05:43:30.637209 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810637206, "job": 746, "event": "table_file_deletion", "file_number": 10243}
>2018-07-11 05:43:30.637478 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10247. Immutable memtables: 0.
>
>2018-07-11 05:43:30.637543 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 747] Syncing log #10244
>2018-07-11 05:43:30.651038 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.637533) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:43:30.651051 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 747] Flushing memtable with next log file: 10247
>
>2018-07-11 05:43:30.651065 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810651058, "job": 747, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 20088}
>2018-07-11 05:43:30.651068 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 747] Level-0 flush table #10248: started
>2018-07-11 05:43:30.693223 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810693210, "cf_name": "default", "job": 747, "event": "table_file_creation", "file_number": 10248, "file_size": 18839, "table_properties": {"data_size": 17773, "index_size": 84, "filter_size": 68, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 17550, "raw_average_value_size": 2193, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:43:30.693233 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 747] Level-0 flush table #10248: 18839 bytes OK
>2018-07-11 05:43:30.743251 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.709735) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10248 started
>2018-07-11 05:43:30.743259 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.743217) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10248: memtable #1 done
>2018-07-11 05:43:30.743262 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.743232) EVENT_LOG_v1 {"time_micros": 1531287810743225, "job": 747, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:43:30.743264 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.743243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:43:30.743282 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 747] Try to delete WAL files size 17724, prev total WAL file size 17724, number of live WAL files 2.
>
>2018-07-11 05:43:30.743386 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:43:30.743433 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:30.743427) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:43:30.743442 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 748] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:43:30.743448 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 747 Base level 0, inputs: [10248(18KB)], [10246(14MB)]
>
>2018-07-11 05:43:30.743460 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810743451, "job": 748, "event": "compaction_started", "files_L0": [10248], "files_L1": [10246], "score": -1, "input_data_size": 15143806}
>2018-07-11 05:43:30.927303 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 748] Generated table #10249: 3318 keys, 15142689 bytes
>2018-07-11 05:43:30.927341 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287810927326, "cf_name": "default", "job": 748, "event": "table_file_creation", "file_number": 10249, "file_size": 15142689, "table_properties": {"data_size": 15026433, "index_size": 66143, "filter_size": 49186, "raw_key_size": 66821, "raw_average_key_size": 20, "raw_value_size": 14932723, "raw_average_value_size": 4500, "num_data_blocks": 1928, "num_entries": 3318, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:43:30.960431 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 748] Compacted 1@0 + 1@1 files to L1 => 15142689 bytes
>2018-07-11 05:43:31.002308 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:31.002238) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.8 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(0.0, 14.4) out(14.4), read-write-amplify(1607.6) write-amplify(803.8) OK, records in: 3323, records dropped: 5
>
>2018-07-11 05:43:31.002317 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:31.002260) EVENT_LOG_v1 {"time_micros": 1531287811002250, "job": 748, "event": "compaction_finished", "compaction_time_micros": 216956, "output_level": 1, "num_output_files": 1, "total_output_size": 15142689, "num_input_records": 3323, "num_output_records": 3318, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:43:31.002413 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287811002408, "job": 748, "event": "table_file_deletion", "file_number": 10248}
>2018-07-11 05:43:31.005474 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287811005471, "job": 748, "event": "table_file_deletion", "file_number": 10246}
>2018-07-11 05:43:37.129798 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217149 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:37.303534 7f29149e4700  1 mon.magna102@0(leader).log v19263 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19263)
>2018-07-11 05:43:43.132456 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217152 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:43.304125 7f29149e4700  1 mon.magna102@0(leader).log v19264 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19264)
>2018-07-11 05:43:49.135000 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217155 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:49.304779 7f29149e4700  1 mon.magna102@0(leader).log v19265 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19265)
>2018-07-11 05:43:50.280531 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10250. Immutable memtables: 0.
>
>2018-07-11 05:43:50.280572 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 749] Syncing log #10247
>2018-07-11 05:43:50.296844 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.280564) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:43:50.296855 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 749] Flushing memtable with next log file: 10250
>
>2018-07-11 05:43:50.296881 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830296873, "job": 749, "event": "flush_started", "num_memtables": 1, "num_entries": 599, "num_deletes": 500, "memory_usage": 242672}
>2018-07-11 05:43:50.296884 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 749] Level-0 flush table #10251: started
>2018-07-11 05:43:50.330709 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830330696, "cf_name": "default", "job": 749, "event": "table_file_creation", "file_number": 10251, "file_size": 223727, "table_properties": {"data_size": 219891, "index_size": 963, "filter_size": 1954, "raw_key_size": 11079, "raw_average_key_size": 20, "raw_value_size": 212041, "raw_average_value_size": 394, "num_data_blocks": 30, "num_entries": 538, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 05:43:50.330719 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 749] Level-0 flush table #10251: 223727 bytes OK
>2018-07-11 05:43:50.380739 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.347265) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10251 started
>2018-07-11 05:43:50.380747 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.380704) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10251: memtable #1 done
>2018-07-11 05:43:50.380750 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.380719) EVENT_LOG_v1 {"time_micros": 1531287830380712, "job": 749, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:43:50.380752 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.380730) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:43:50.380771 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 749] Try to delete WAL files size 226742, prev total WAL file size 226742, number of live WAL files 2.
>
>2018-07-11 05:43:50.380909 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:43:50.380960 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.380953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 05:43:50.380969 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 750] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:43:50.380975 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 749 Base level 0, inputs: [10251(218KB)], [10249(14MB)]
>
>2018-07-11 05:43:50.380988 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830380979, "job": 750, "event": "compaction_started", "files_L0": [10251], "files_L1": [10249], "score": -1, "input_data_size": 15366416}
>2018-07-11 05:43:50.539633 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 750] Generated table #10252: 2848 keys, 13077621 bytes
>2018-07-11 05:43:50.539675 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830539648, "cf_name": "default", "job": 750, "event": "table_file_creation", "file_number": 10252, "file_size": 13077621, "table_properties": {"data_size": 12976416, "index_size": 57728, "filter_size": 42550, "raw_key_size": 57194, "raw_average_key_size": 20, "raw_value_size": 12895325, "raw_average_value_size": 4527, "num_data_blocks": 1693, "num_entries": 2848, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:43:50.572976 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 750] Compacted 1@0 + 1@1 files to L1 => 13077621 bytes
>2018-07-11 05:43:50.606520 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.606453) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 80.0 rd, 68.1 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.4) out(12.5), read-write-amplify(127.1) write-amplify(58.5) OK, records in: 3856, records dropped: 1008
>
>2018-07-11 05:43:50.606529 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:43:50.606475) EVENT_LOG_v1 {"time_micros": 1531287830606466, "job": 750, "event": "compaction_finished", "compaction_time_micros": 191972, "output_level": 1, "num_output_files": 1, "total_output_size": 13077621, "num_input_records": 3856, "num_output_records": 2848, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:43:50.606654 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830606650, "job": 750, "event": "table_file_deletion", "file_number": 10251}
>2018-07-11 05:43:50.609514 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287830609511, "job": 750, "event": "table_file_deletion", "file_number": 10249}
>2018-07-11 05:43:50.609630 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:43:55.137740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217158 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:43:55.325499 7f29149e4700  1 mon.magna102@0(leader).log v19266 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19266)
>2018-07-11 05:44:00.238985 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217161 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:00.325821 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:44:00.367660 7f29149e4700  1 mon.magna102@0(leader).log v19267 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19267)
>2018-07-11 05:44:05.239168 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217164 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:06.301600 7f29149e4700  1 mon.magna102@0(leader).log v19268 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19268)
>2018-07-11 05:44:11.144705 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217167 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:11.302252 7f29149e4700  1 mon.magna102@0(leader).log v19269 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19269)
>2018-07-11 05:44:17.147467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217170 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:17.294506 7f29149e4700  1 mon.magna102@0(leader).log v19270 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19270)
>2018-07-11 05:44:27.151743 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217173 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:27.303781 7f29149e4700  1 mon.magna102@0(leader).log v19271 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19271)
>2018-07-11 05:44:35.240172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217176 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:36.287816 7f29149e4700  1 mon.magna102@0(leader).log v19272 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19272)
>2018-07-11 05:44:40.240366 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217179 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:40.355003 7f29149e4700  1 mon.magna102@0(leader).log v19273 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19273)
>2018-07-11 05:44:45.240576 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217182 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:46.297291 7f29149e4700  1 mon.magna102@0(leader).log v19274 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19274)
>2018-07-11 05:44:51.162119 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217185 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:51.324154 7f29149e4700  1 mon.magna102@0(leader).log v19275 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19275)
>2018-07-11 05:44:59.165569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217188 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:44:59.316547 7f29149e4700  1 mon.magna102@0(leader).log v19276 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19276)
>2018-07-11 05:45:00.325994 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:45:05.241287 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217191 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:06.300650 7f29149e4700  1 mon.magna102@0(leader).log v19277 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19277)
>2018-07-11 05:45:13.171787 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217194 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:13.326421 7f29149e4700  1 mon.magna102@0(leader).log v19278 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19278)
>2018-07-11 05:45:19.174296 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217197 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:19.327020 7f29149e4700  1 mon.magna102@0(leader).log v19279 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19279)
>2018-07-11 05:45:25.242006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217200 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:25.364665 7f29149e4700  1 mon.magna102@0(leader).log v19280 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19280)
>2018-07-11 05:45:30.242224 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217203 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:30.336434 7f29149e4700  1 mon.magna102@0(leader).log v19281 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19281)
>2018-07-11 05:45:40.242639 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217206 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:40.345742 7f29149e4700  1 mon.magna102@0(leader).log v19282 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19282)
>2018-07-11 05:45:45.242872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217209 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:45.364301 7f29149e4700  1 mon.magna102@0(leader).log v19283 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19283)
>2018-07-11 05:45:50.243108 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217212 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:50.323085 7f29149e4700  1 mon.magna102@0(leader).log v19284 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19284)
>2018-07-11 05:45:55.243355 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217215 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:45:55.357084 7f29149e4700  1 mon.magna102@0(leader).log v19285 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19285)
>2018-07-11 05:46:00.326173 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:46:01.192461 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217218 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:01.349373 7f29149e4700  1 mon.magna102@0(leader).log v19286 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19286)
>2018-07-11 05:46:07.195961 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217221 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:07.350531 7f29149e4700  1 mon.magna102@0(leader).log v19287 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19287)
>2018-07-11 05:46:10.214471 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:10.214496 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1556624691' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:10.326191 7f29149e4700  1 mon.magna102@0(leader).log v19288 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19288)
>2018-07-11 05:46:13.198962 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217224 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:13.351679 7f29149e4700  1 mon.magna102@0(leader).log v19289 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19289)
>2018-07-11 05:46:14.160793 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:14.160817 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2593100269' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:14.418523 7f29149e4700  1 mon.magna102@0(leader).log v19290 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19290)
>2018-07-11 05:46:15.743049 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:15.743072 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/573821042' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:16.560468 7f29149e4700  1 mon.magna102@0(leader).log v19291 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19291)
>2018-07-11 05:46:17.446524 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:17.446547 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2097083849' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:17.694307 7f29149e4700  1 mon.magna102@0(leader).log v19292 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19292)
>2018-07-11 05:46:21.303231 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:21.303255 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1494174313' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:22.336441 7f29149e4700  1 mon.magna102@0(leader).log v19293 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19293)
>2018-07-11 05:46:23.336458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217227 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:23.419877 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:46:23.419898 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/583654655' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:46:23.536881 7f29149e4700  1 mon.magna102@0(leader).log v19294 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19294)
>2018-07-11 05:46:30.287057 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217230 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:30.413677 7f29149e4700  1 mon.magna102@0(leader).log v19295 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19295)
>2018-07-11 05:46:35.287299 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217233 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:35.414471 7f29149e4700  1 mon.magna102@0(leader).log v19296 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19296)
>2018-07-11 05:46:40.287526 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217236 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:40.373263 7f29149e4700  1 mon.magna102@0(leader).log v19297 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19297)
>2018-07-11 05:46:45.287764 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217239 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:45.415573 7f29149e4700  1 mon.magna102@0(leader).log v19298 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19298)
>2018-07-11 05:46:53.217400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217242 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:46:53.391390 7f29149e4700  1 mon.magna102@0(leader).log v19299 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19299)
>2018-07-11 05:47:00.288363 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217245 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:00.341688 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:47:00.383699 7f29149e4700  1 mon.magna102@0(leader).log v19300 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19300)
>2018-07-11 05:47:05.288593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217248 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:05.409383 7f29149e4700  1 mon.magna102@0(leader).log v19301 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19301)
>2018-07-11 05:47:10.859705 7f29149e4700  1 mon.magna102@0(leader).log v19302 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19302)
>2018-07-11 05:47:10.878364 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd lspools"} v 0) v1
>2018-07-11 05:47:10.878392 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3519689357' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch
>2018-07-11 05:47:11.926602 7f29149e4700  1 mon.magna102@0(leader).log v19303 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19303)
>2018-07-11 05:47:13.226420 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217251 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:13.377200 7f29149e4700  1 mon.magna102@0(leader).log v19304 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19304)
>2018-07-11 05:47:19.228978 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217254 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:19.386084 7f29149e4700  1 mon.magna102@0(leader).log v19305 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19305)
>2018-07-11 05:47:25.289417 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217257 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:25.404736 7f29149e4700  1 mon.magna102@0(leader).log v19306 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19306)
>2018-07-11 05:47:33.235173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217260 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:33.388789 7f29149e4700  1 mon.magna102@0(leader).log v19307 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19307)
>2018-07-11 05:47:39.237600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217263 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:39.389491 7f29149e4700  1 mon.magna102@0(leader).log v19308 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19308)
>2018-07-11 05:47:45.290176 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217266 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:45.406788 7f29149e4700  1 mon.magna102@0(leader).log v19309 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19309)
>2018-07-11 05:47:53.243828 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217269 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:47:53.399174 7f29149e4700  1 mon.magna102@0(leader).log v19310 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19310)
>2018-07-11 05:48:00.290771 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217272 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:00.349692 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8787 MB, avail 861 GB
>2018-07-11 05:48:00.383225 7f29149e4700  1 mon.magna102@0(leader).log v19311 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19311)
>2018-07-11 05:48:05.291020 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217275 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:05.408858 7f29149e4700  1 mon.magna102@0(leader).log v19312 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19312)
>2018-07-11 05:48:11.251564 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217278 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:11.426149 7f29149e4700  1 mon.magna102@0(leader).log v19313 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19313)
>2018-07-11 05:48:17.254299 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217281 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:17.427959 7f29149e4700  1 mon.magna102@0(leader).log v19314 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19314)
>2018-07-11 05:48:25.257780 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217284 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:25.412096 7f29149e4700  1 mon.magna102@0(leader).log v19315 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19315)
>2018-07-11 05:48:33.261406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217287 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:33.412911 7f29149e4700  1 mon.magna102@0(leader).log v19316 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19316)
>2018-07-11 05:48:40.295593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217290 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:40.396825 7f29149e4700  1 mon.magna102@0(leader).log v19317 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19317)
>2018-07-11 05:48:45.305416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217293 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:45.406285 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10253. Immutable memtables: 0.
>
>2018-07-11 05:48:45.406328 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 751] Syncing log #10250
>2018-07-11 05:48:45.422597 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.406318) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:48:45.422607 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 751] Flushing memtable with next log file: 10253
>
>2018-07-11 05:48:45.422634 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125422626, "job": 751, "event": "flush_started", "num_memtables": 1, "num_entries": 1708, "num_deletes": 251, "memory_usage": 3219864}
>2018-07-11 05:48:45.422637 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 751] Level-0 flush table #10254: started
>2018-07-11 05:48:45.523362 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125523347, "cf_name": "default", "job": 751, "event": "table_file_creation", "file_number": 10254, "file_size": 3108670, "table_properties": {"data_size": 3083908, "index_size": 13434, "filter_size": 10404, "raw_key_size": 14635, "raw_average_key_size": 20, "raw_value_size": 3064384, "raw_average_value_size": 4303, "num_data_blocks": 404, "num_entries": 712, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:48:45.523374 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 751] Level-0 flush table #10254: 3108670 bytes OK
>2018-07-11 05:48:45.565112 7f29149e4700  1 mon.magna102@0(leader).log v19318 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19318)
>2018-07-11 05:48:45.606627 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.564871) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10254 started
>2018-07-11 05:48:45.606636 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.606588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10254: memtable #1 done
>2018-07-11 05:48:45.606639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.606603) EVENT_LOG_v1 {"time_micros": 1531288125606597, "job": 751, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:48:45.606641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.606617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:48:45.606647 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 751] Try to delete WAL files size 3185747, prev total WAL file size 3204797, number of live WAL files 2.
>
>2018-07-11 05:48:45.607444 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:48:45.607627 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 05:48:45.607636 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 50152.5 total, 815.5 interval
>Cumulative writes: 70K writes, 336K keys, 70K commit groups, 1.0 writes per commit group, ingest: 0.56 GB, 0.01 MB/s
>Cumulative WAL: 70K writes, 70K syncs, 1.00 writes per sync, written: 0.56 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1104 writes, 5496 keys, 1104 commit groups, 1.0 writes per commit group, ingest: 8.30 MB, 0.01 MB/s
>Interval WAL: 1104 writes, 1104 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.96 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        31       376    0.083       0      0
>  L1      1/1   12.47 MB   0.0      5.9     0.5      5.4       5.4     -0.0       0.0  10.9     72.3     66.2        83       374    0.222   1323K   156K
> Sum      2/2   15.44 MB   0.0      5.9     0.5      5.4       5.9      0.5       0.0  11.9     52.5     52.5       114       750    0.152   1323K   156K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.8     53.5     53.4         2        12    0.146     22K   2880
>Uptime(secs): 50152.5 total, 50152.5 interval
>Flush(GB): cumulative 0.494, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.86 GB write, 0.12 MB/s write, 5.86 GB read, 0.12 MB/s read, 114.2 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.09 GB read, 0.00 MB/s read, 1.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.96 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.2        31       376    0.083       0      0
>  L1      1/1   12.47 MB   0.0      5.9     0.5      5.4       5.4     -0.0       0.0  10.9     72.3     66.2        83       374    0.222   1323K   156K
> Sum      2/2   15.44 MB   0.0      5.9     0.5      5.4       5.9      0.5       0.0  11.9     52.5     52.5       114       750    0.152   1323K   156K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 50152.5 total, 50152.5 interval
>Flush(GB): cumulative 0.494, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.86 GB write, 0.12 MB/s write, 5.86 GB read, 0.12 MB/s read, 114.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 05:48:45.607659 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.607653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:48:45.607664 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 752] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:48:45.607670 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 751 Base level 0, inputs: [10254(3035KB)], [10252(12MB)]
>
>2018-07-11 05:48:45.607682 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125607673, "job": 752, "event": "compaction_started", "files_L0": [10254], "files_L1": [10252], "score": -1, "input_data_size": 16186291}
>2018-07-11 05:48:45.782432 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 752] Generated table #10255: 3050 keys, 14180201 bytes
>2018-07-11 05:48:45.782483 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125782467, "cf_name": "default", "job": 752, "event": "table_file_creation", "file_number": 10255, "file_size": 14180201, "table_properties": {"data_size": 14069938, "index_size": 63173, "filter_size": 46163, "raw_key_size": 61582, "raw_average_key_size": 20, "raw_value_size": 13982094, "raw_average_value_size": 4584, "num_data_blocks": 1845, "num_entries": 3050, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:48:45.815585 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 752] Compacted 1@0 + 1@1 files to L1 => 14180201 bytes
>2018-07-11 05:48:45.849131 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.849064) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.9 rd, 68.2 wr, level 1, files in(1, 1) out(1) MB in(3.0, 12.5) out(13.5), read-write-amplify(9.8) write-amplify(4.6) OK, records in: 3560, records dropped: 510
>
>2018-07-11 05:48:45.849140 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:48:45.849085) EVENT_LOG_v1 {"time_micros": 1531288125849076, "job": 752, "event": "compaction_finished", "compaction_time_micros": 207885, "output_level": 1, "num_output_files": 1, "total_output_size": 14180201, "num_input_records": 3560, "num_output_records": 3050, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:48:45.849803 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125849798, "job": 752, "event": "table_file_deletion", "file_number": 10254}
>2018-07-11 05:48:45.852347 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288125852344, "job": 752, "event": "table_file_deletion", "file_number": 10252}
>2018-07-11 05:48:50.305660 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217296 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:50.391729 7f29149e4700  1 mon.magna102@0(leader).log v19319 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19319)
>2018-07-11 05:48:57.272069 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217299 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:48:57.425837 7f29149e4700  1 mon.magna102@0(leader).log v19320 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19320)
>2018-07-11 05:49:00.349856 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:49:05.309004 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217302 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:05.427831 7f29149e4700  1 mon.magna102@0(leader).log v19321 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19321)
>2018-07-11 05:49:10.309229 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217305 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:10.411620 7f29149e4700  1 mon.magna102@0(leader).log v19322 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19322)
>2018-07-11 05:49:15.311951 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217308 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:15.429019 7f29149e4700  1 mon.magna102@0(leader).log v19323 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19323)
>2018-07-11 05:49:19.706239 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 05:49:19.706273 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2971492523' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 05:49:20.421144 7f29149e4700  1 mon.magna102@0(leader).log v19324 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19324)
>2018-07-11 05:49:21.421159 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217311 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:21.571519 7f29149e4700  1 mon.magna102@0(leader).log v19325 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19325)
>2018-07-11 05:49:27.285368 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217314 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:27.438783 7f29149e4700  1 mon.magna102@0(leader).log v19326 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19326)
>2018-07-11 05:49:35.288937 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217317 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:35.439516 7f29149e4700  1 mon.magna102@0(leader).log v19327 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19327)
>2018-07-11 05:49:40.848203 7f29149e4700  1 mon.magna102@0(leader).log v19328 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19328)
>2018-07-11 05:49:43.292579 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217320 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:43.448769 7f29149e4700  1 mon.magna102@0(leader).log v19329 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19329)
>2018-07-11 05:49:50.332274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217323 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:50.441075 7f29149e4700  1 mon.magna102@0(leader).log v19330 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19330)
>2018-07-11 05:49:59.299430 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217329 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:49:59.476637 7f29149e4700  1 mon.magna102@0(leader).log v19331 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19331)
>2018-07-11 05:50:00.350037 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:50:09.304172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217332 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:09.452703 7f29149e4700  1 mon.magna102@0(leader).log v19332 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19332)
>2018-07-11 05:50:15.344070 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217335 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:15.461679 7f29149e4700  1 mon.magna102@0(leader).log v19333 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19333)
>2018-07-11 05:50:20.344294 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217338 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:20.437135 7f29149e4700  1 mon.magna102@0(leader).log v19334 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19334)
>2018-07-11 05:50:29.312871 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217341 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:29.463011 7f29149e4700  1 mon.magna102@0(leader).log v19335 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19335)
>2018-07-11 05:50:35.355017 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217344 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:35.472130 7f29149e4700  1 mon.magna102@0(leader).log v19336 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19336)
>2018-07-11 05:50:40.355238 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217347 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:40.464228 7f29149e4700  1 mon.magna102@0(leader).log v19337 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19337)
>2018-07-11 05:50:49.321406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217353 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:49.473486 7f29149e4700  1 mon.magna102@0(leader).log v19338 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19338)
>2018-07-11 05:50:57.325020 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217356 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:50:57.475422 7f29149e4700  1 mon.magna102@0(leader).log v19339 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19339)
>2018-07-11 05:51:00.350222 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8788 MB, avail 861 GB
>2018-07-11 05:51:03.327503 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217359 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:03.484457 7f29149e4700  1 mon.magna102@0(leader).log v19340 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19340)
>2018-07-11 05:51:11.331031 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217362 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:11.493514 7f29149e4700  1 mon.magna102@0(leader).log v19341 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19341)
>2018-07-11 05:51:19.334511 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217365 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:19.485924 7f29149e4700  1 mon.magna102@0(leader).log v19342 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19342)
>2018-07-11 05:51:25.377254 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217368 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:25.511525 7f29149e4700  1 mon.magna102@0(leader).log v19343 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19343)
>2018-07-11 05:51:30.377478 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217371 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:30.503649 7f29149e4700  1 mon.magna102@0(leader).log v19344 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19344)
>2018-07-11 05:51:37.342400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217374 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:37.512739 7f29149e4700  1 mon.magna102@0(leader).log v19345 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19345)
>2018-07-11 05:51:45.388078 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217377 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:45.513491 7f29149e4700  1 mon.magna102@0(leader).log v19346 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19346)
>2018-07-11 05:51:50.388323 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217380 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:50.464376 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10256. Immutable memtables: 0.
>
>2018-07-11 05:51:50.464422 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 753] Syncing log #10253
>2018-07-11 05:51:50.481872 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.464412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:51:50.481883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 753] Flushing memtable with next log file: 10256
>
>2018-07-11 05:51:50.481898 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310481890, "job": 753, "event": "flush_started", "num_memtables": 1, "num_entries": 1131, "num_deletes": 250, "memory_usage": 1959664}
>2018-07-11 05:51:50.481901 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 753] Level-0 flush table #10257: started
>2018-07-11 05:51:50.557759 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310557738, "cf_name": "default", "job": 753, "event": "table_file_creation", "file_number": 10257, "file_size": 1301640, "table_properties": {"data_size": 1290926, "index_size": 5147, "filter_size": 4647, "raw_key_size": 9470, "raw_average_key_size": 21, "raw_value_size": 1281104, "raw_average_value_size": 2885, "num_data_blocks": 159, "num_entries": 444, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 05:51:50.557775 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 753] Level-0 flush table #10257: 1301640 bytes OK
>2018-07-11 05:51:50.591110 7f29149e4700  1 mon.magna102@0(leader).log v19347 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19347)
>2018-07-11 05:51:50.624359 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.590861) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10257 started
>2018-07-11 05:51:50.624368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.624311) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10257: memtable #1 done
>2018-07-11 05:51:50.624370 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.624327) EVENT_LOG_v1 {"time_micros": 1531288310624320, "job": 753, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:51:50.624372 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.624346) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:51:50.624381 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 753] Try to delete WAL files size 1936729, prev total WAL file size 1955747, number of live WAL files 2.
>
>2018-07-11 05:51:50.624927 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:51:50.624980 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.624973) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:51:50.624990 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 754] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:51:50.624996 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 753 Base level 0, inputs: [10257(1271KB)], [10255(13MB)]
>
>2018-07-11 05:51:50.625009 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310624999, "job": 754, "event": "compaction_started", "files_L0": [10257], "files_L1": [10255], "score": -1, "input_data_size": 15481841}
>2018-07-11 05:51:50.757627 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd reweightn", "weights": "{'10':'58984'}"} v 0) v1
>2018-07-11 05:51:50.757675 7f29191ed700  0 log_channel(audit) log [INF] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd reweightn", "weights": "{'10':'58984'}"}]: dispatch
>2018-07-11 05:51:50.816703 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 754] Generated table #10258: 3072 keys, 14355913 bytes
>2018-07-11 05:51:50.816732 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310816717, "cf_name": "default", "job": 754, "event": "table_file_creation", "file_number": 10258, "file_size": 14355913, "table_properties": {"data_size": 14245693, "index_size": 62775, "filter_size": 46518, "raw_key_size": 61735, "raw_average_key_size": 20, "raw_value_size": 14157887, "raw_average_value_size": 4608, "num_data_blocks": 1840, "num_entries": 3072, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:51:50.849985 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 754] Compacted 1@0 + 1@1 files to L1 => 14355913 bytes
>2018-07-11 05:51:50.883514 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.883448) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.8 rd, 63.8 wr, level 1, files in(1, 1) out(1) MB in(1.2, 13.5) out(13.7), read-write-amplify(22.9) write-amplify(11.0) OK, records in: 3494, records dropped: 422
>
>2018-07-11 05:51:50.883523 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.883469) EVENT_LOG_v1 {"time_micros": 1531288310883459, "job": 754, "event": "compaction_finished", "compaction_time_micros": 224960, "output_level": 1, "num_output_files": 1, "total_output_size": 14355913, "num_input_records": 3494, "num_output_records": 3072, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:51:50.883878 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310883874, "job": 754, "event": "table_file_deletion", "file_number": 10257}
>2018-07-11 05:51:50.886522 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310886519, "job": 754, "event": "table_file_deletion", "file_number": 10255}
>2018-07-11 05:51:50.886838 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10259. Immutable memtables: 0.
>
>2018-07-11 05:51:50.886876 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 755] Syncing log #10256
>2018-07-11 05:51:50.900461 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.886868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:51:50.900472 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 755] Flushing memtable with next log file: 10259
>
>2018-07-11 05:51:50.900499 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310900491, "job": 755, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21368}
>2018-07-11 05:51:50.900502 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 755] Level-0 flush table #10260: started
>2018-07-11 05:51:50.942565 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310942553, "cf_name": "default", "job": 755, "event": "table_file_creation", "file_number": 10260, "file_size": 20137, "table_properties": {"data_size": 19067, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18844, "raw_average_value_size": 2355, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:51:50.942575 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 755] Level-0 flush table #10260: 20137 bytes OK
>2018-07-11 05:51:50.992661 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.959162) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10260 started
>2018-07-11 05:51:50.992669 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.992627) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10260: memtable #1 done
>2018-07-11 05:51:50.992671 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.992641) EVENT_LOG_v1 {"time_micros": 1531288310992635, "job": 755, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:51:50.992673 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.992652) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:51:50.992691 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 755] Try to delete WAL files size 19018, prev total WAL file size 19018, number of live WAL files 2.
>
>2018-07-11 05:51:50.992790 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:51:50.992836 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:50.992830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 05:51:50.992845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 756] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:51:50.992851 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 755 Base level 0, inputs: [10260(19KB)], [10258(13MB)]
>
>2018-07-11 05:51:50.992863 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288310992854, "job": 756, "event": "compaction_started", "files_L0": [10260], "files_L1": [10258], "score": -1, "input_data_size": 14376050}
>2018-07-11 05:51:51.168392 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 756] Generated table #10261: 3075 keys, 14374933 bytes
>2018-07-11 05:51:51.168437 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288311168409, "cf_name": "default", "job": 756, "event": "table_file_creation", "file_number": 10261, "file_size": 14374933, "table_properties": {"data_size": 14264587, "index_size": 62844, "filter_size": 46575, "raw_key_size": 61796, "raw_average_key_size": 20, "raw_value_size": 14176691, "raw_average_value_size": 4610, "num_data_blocks": 1842, "num_entries": 3075, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:51:51.201588 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 756] Compacted 1@0 + 1@1 files to L1 => 14374933 bytes
>2018-07-11 05:51:51.235123 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:51.235056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 68.9 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 13.7) out(13.7), read-write-amplify(1427.8) write-amplify(713.9) OK, records in: 3080, records dropped: 5
>
>2018-07-11 05:51:51.235132 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:51:51.235078) EVENT_LOG_v1 {"time_micros": 1531288311235069, "job": 756, "event": "compaction_finished", "compaction_time_micros": 208708, "output_level": 1, "num_output_files": 1, "total_output_size": 14374933, "num_input_records": 3080, "num_output_records": 3075, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:51:51.235237 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288311235233, "job": 756, "event": "table_file_deletion", "file_number": 10260}
>2018-07-11 05:51:51.237757 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288311237754, "job": 756, "event": "table_file_deletion", "file_number": 10258}
>2018-07-11 05:51:51.660768 7f29149e4700  1 mon.magna102@0(leader).log v19348 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19348)
>2018-07-11 05:51:51.727744 7f29149e4700  1 mon.magna102@0(leader).osd e462 e462: 12 total, 12 up, 12 in
>2018-07-11 05:51:51.761337 7f29149e4700  0 log_channel(audit) log [INF] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd='[{"prefix": "osd reweightn", "weights": "{'10':'58984'}"}]': finished
>2018-07-11 05:51:51.761397 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e462: 12 total, 12 up, 12 in
>2018-07-11 05:51:52.794612 7f29149e4700  1 mon.magna102@0(leader).log v19349 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19349)
>2018-07-11 05:51:55.978470 7f29149e4700  1 mon.magna102@0(leader).osd e463 e463: 12 total, 12 up, 12 in
>2018-07-11 05:51:56.012199 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e463: 12 total, 12 up, 12 in
>2018-07-11 05:51:57.087020 7f29149e4700  1 mon.magna102@0(leader).log v19350 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19350)
>2018-07-11 05:51:58.087112 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217383 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:51:58.245770 7f29149e4700  1 mon.magna102@0(leader).log v19351 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19351)
>2018-07-11 05:52:00.350375 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:52:00.463420 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:00.463443 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1310553214' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:00.579527 7f29149e4700  1 mon.magna102@0(leader).log v19352 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19352)
>2018-07-11 05:52:01.579555 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 507/24217386 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:01.713119 7f29149e4700  1 mon.magna102@0(leader).log v19353 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19353)
>2018-07-11 05:52:02.032562 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:02.032585 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3639487968' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:02.780135 7f29149e4700  1 mon.magna102@0(leader).log v19354 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19354)
>2018-07-11 05:52:03.465226 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:03.465249 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4139489478' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:03.846887 7f29149e4700  1 mon.magna102@0(leader).log v19355 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19355)
>2018-07-11 05:52:04.733829 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:04.733851 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3052201132' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:04.980621 7f29149e4700  1 mon.magna102@0(leader).log v19356 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19356)
>2018-07-11 05:52:05.431074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217386 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:06.021640 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:06.021665 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3987438162' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:06.047561 7f29149e4700  1 mon.magna102@0(leader).log v19357 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19357)
>2018-07-11 05:52:07.122735 7f29149e4700  1 mon.magna102@0(leader).log v19358 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19358)
>2018-07-11 05:52:07.394137 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:07.394160 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2658904780' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:08.157519 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217389 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:08.190920 7f29149e4700  1 mon.magna102@0(leader).log v19359 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19359)
>2018-07-11 05:52:09.057509 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:09.057532 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3420720066' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:09.258080 7f29149e4700  1 mon.magna102@0(leader).log v19360 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19360)
>2018-07-11 05:52:10.431332 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217389 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:10.645074 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:10.645097 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1392880561' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:11.467009 7f29149e4700  1 mon.magna102@0(leader).log v19361 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19361)
>2018-07-11 05:52:12.033551 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:12.033575 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/203827115' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:12.542150 7f29149e4700  1 mon.magna102@0(leader).log v19362 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19362)
>2018-07-11 05:52:13.542216 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217392 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:13.633602 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:52:13.633626 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2604968161' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:52:13.675704 7f29191ed700  1 mon.magna102@0(leader).log v19362 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 1 entries (version 19362)
>2018-07-11 05:52:13.709269 7f29149e4700  1 mon.magna102@0(leader).log v19363 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19363)
>2018-07-11 05:52:13.709316 7f29149e4700  1 mon.magna102@0(leader).log v19363 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 1 entries (version 19363)
>2018-07-11 05:52:15.431616 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217392 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:15.559600 7f29149e4700  1 mon.magna102@0(leader).log v19364 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19364)
>2018-07-11 05:52:15.559645 7f29149e4700  1 mon.magna102@0(leader).log v19364 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 1 entries (version 19364)
>2018-07-11 05:52:23.364600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217395 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:23.364647 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217395 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:23.535439 7f29149e4700  1 mon.magna102@0(leader).log v19365 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19365)
>2018-07-11 05:52:23.535485 7f29149e4700  1 mon.magna102@0(leader).log v19365 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19365)
>2018-07-11 05:52:31.368145 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217398 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:31.368200 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217398 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:31.544558 7f29149e4700  1 mon.magna102@0(leader).log v19366 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19366)
>2018-07-11 05:52:31.544605 7f29149e4700  1 mon.magna102@0(leader).log v19366 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19366)
>2018-07-11 05:52:39.371702 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217404 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:39.371747 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217404 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:39.537056 7f29149e4700  1 mon.magna102@0(leader).log v19367 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19367)
>2018-07-11 05:52:39.537102 7f29149e4700  1 mon.magna102@0(leader).log v19367 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19367)
>2018-07-11 05:52:49.376150 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217407 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:49.376205 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217407 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:49.521358 7f29149e4700  1 mon.magna102@0(leader).log v19368 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19368)
>2018-07-11 05:52:49.521405 7f29149e4700  1 mon.magna102@0(leader).log v19368 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19368)
>2018-07-11 05:52:55.433130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217410 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:52:55.433185 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217410 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:52:55.548850 7f29149e4700  1 mon.magna102@0(leader).log v19369 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19369)
>2018-07-11 05:52:55.548896 7f29149e4700  1 mon.magna102@0(leader).log v19369 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19369)
>2018-07-11 05:53:00.350567 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:53:01.382208 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217413 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:01.382256 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217413 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:01.540973 7f29149e4700  1 mon.magna102@0(leader).log v19370 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19370)
>2018-07-11 05:53:01.541019 7f29149e4700  1 mon.magna102@0(leader).log v19370 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19370)
>2018-07-11 05:53:09.385713 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217416 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:09.385759 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 499/24217416 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:09.533374 7f29149e4700  1 mon.magna102@0(leader).log v19371 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19371)
>2018-07-11 05:53:09.533421 7f29149e4700  1 mon.magna102@0(leader).log v19371 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19371)
>2018-07-11 05:53:15.434152 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217419 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:15.434202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217419 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:15.580402 7f29149e4700  1 mon.magna102@0(leader).log v19372 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19372)
>2018-07-11 05:53:15.580466 7f29149e4700  1 mon.magna102@0(leader).log v19372 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19372)
>2018-07-11 05:53:20.434409 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217422 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:20.434458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217422 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:20.526880 7f29149e4700  1 mon.magna102@0(leader).log v19373 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19373)
>2018-07-11 05:53:20.526926 7f29149e4700  1 mon.magna102@0(leader).log v19373 check_sub sending message to client.? 10.8.128.102:0/2604968161 with 2 entries (version 19373)
>2018-07-11 05:53:24.436101 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:24.436124 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2505452355' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:24.527348 7f29149e4700  1 mon.magna102@0(leader).log v19374 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19374)
>2018-07-11 05:53:25.434695 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217425 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:25.434740 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217425 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:25.602662 7f29149e4700  1 mon.magna102@0(leader).log v19375 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19375)
>2018-07-11 05:53:25.917833 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:25.917856 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2443401689' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:26.736247 7f29149e4700  1 mon.magna102@0(leader).log v19376 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19376)
>2018-07-11 05:53:27.236309 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:27.236332 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2276815736' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:27.869872 7f29149e4700  1 mon.magna102@0(leader).log v19377 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19377)
>2018-07-11 05:53:28.589918 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:28.589940 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1435193153' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:28.945119 7f29149e4700  1 mon.magna102@0(leader).log v19378 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19378)
>2018-07-11 05:53:30.394315 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:30.394338 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2332274736' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:31.112248 7f29149e4700  1 mon.magna102@0(leader).log v19379 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19379)
>2018-07-11 05:53:31.767728 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:31.767751 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2861794594' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:32.270880 7f29149e4700  1 mon.magna102@0(leader).log v19380 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19380)
>2018-07-11 05:53:33.099234 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:53:33.099257 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2911697005' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:53:33.362750 7f29149e4700  1 mon.magna102@0(leader).log v19381 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19381)
>2018-07-11 05:53:33.397686 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217428 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:33.397733 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217428 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:34.496452 7f29149e4700  1 mon.magna102@0(leader).log v19382 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19382)
>2018-07-11 05:53:39.400390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217431 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:39.400436 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217431 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:39.573426 7f29149e4700  1 mon.magna102@0(leader).log v19383 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19383)
>2018-07-11 05:53:45.440351 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217434 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:45.440384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217434 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:45.565705 7f29149e4700  1 mon.magna102@0(leader).log v19384 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19384)
>2018-07-11 05:53:53.406553 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217437 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:53:53.406600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217437 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:53:53.558215 7f29149e4700  1 mon.magna102@0(leader).log v19385 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19385)
>2018-07-11 05:53:58.867162 7f29191ed700  0 log_channel(cluster) log [INF] : osd.8 marked itself down
>2018-07-11 05:53:58.868408 7f29191ed700  0 log_channel(cluster) log [INF] : osd.5 marked itself down
>2018-07-11 05:53:58.868478 7f29191ed700  0 log_channel(cluster) log [INF] : osd.4 marked itself down
>2018-07-11 05:53:58.868591 7f29191ed700  0 log_channel(cluster) log [INF] : osd.0 marked itself down
>2018-07-11 05:53:58.875164 7f29191ed700  0 log_channel(cluster) log [INF] : osd.9 marked itself down
>2018-07-11 05:53:58.875231 7f29191ed700  0 log_channel(cluster) log [INF] : osd.10 marked itself down
>2018-07-11 05:53:58.879474 7f29191ed700  0 log_channel(cluster) log [INF] : osd.1 marked itself down
>2018-07-11 05:53:58.879573 7f29191ed700  0 log_channel(cluster) log [INF] : osd.3 marked itself down
>2018-07-11 05:53:58.879628 7f29191ed700  0 log_channel(cluster) log [INF] : osd.2 marked itself down
>2018-07-11 05:53:58.918313 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 9 osds down (OSD_DOWN)
>2018-07-11 05:53:58.918348 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 4 hosts (9 osds) down (OSD_HOST_DOWN)
>2018-07-11 05:53:58.992366 7f29149e4700  1 mon.magna102@0(leader).osd e464 e464: 12 total, 3 up, 12 in
>2018-07-11 05:53:59.026077 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e464: 12 total, 3 up, 12 in
>2018-07-11 05:53:59.092598 7f29149e4700  1 mon.magna102@0(leader).log v19386 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 9 entries (version 19386)
>2018-07-11 05:54:00.226267 7f29149e4700  1 mon.magna102@0(leader).log v19387 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19387)
>2018-07-11 05:54:00.350713 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:54:04.830355 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:54:04.830382 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4184507155' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:54:04.831669 7f29191ed700  1 mon.magna102@0(leader).log v19387 check_sub sending message to client.? 10.8.128.102:0/4184507155 with 1 entries (version 19387)
>2018-07-11 05:54:04.943375 7f29149e4700  1 mon.magna102@0(leader).log v19388 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19388)
>2018-07-11 05:54:04.943395 7f29149e4700  1 mon.magna102@0(leader).log v19388 check_sub sending message to client.? 10.8.128.102:0/4184507155 with 0 entries (version 19388)
>2018-07-11 05:54:23.226373 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:54:23.226399 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3303430193' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:54:23.227779 7f29191ed700  1 mon.magna102@0(leader).log v19388 check_sub sending message to client.? 10.8.128.102:0/3303430193 with 1 entries (version 19388)
>2018-07-11 05:54:23.344774 7f29149e4700  1 mon.magna102@0(leader).log v19389 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19389)
>2018-07-11 05:54:23.344808 7f29149e4700  1 mon.magna102@0(leader).log v19389 check_sub sending message to client.? 10.8.128.102:0/3303430193 with 0 entries (version 19389)
>2018-07-11 05:54:31.064457 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:54:31.064479 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2667207926' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:54:31.180330 7f29149e4700  1 mon.magna102@0(leader).log v19390 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19390)
>2018-07-11 05:54:32.704810 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:54:32.704833 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3136937481' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:54:33.322368 7f29149e4700  1 mon.magna102@0(leader).log v19391 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19391)
>2018-07-11 05:54:33.389827 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10262. Immutable memtables: 0.
>
>2018-07-11 05:54:33.389884 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 757] Syncing log #10259
>2018-07-11 05:54:33.406012 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.389874) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:54:33.406023 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 757] Flushing memtable with next log file: 10262
>
>2018-07-11 05:54:33.406050 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473406030, "job": 757, "event": "flush_started", "num_memtables": 1, "num_entries": 1201, "num_deletes": 251, "memory_usage": 2230016}
>2018-07-11 05:54:33.406054 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 757] Level-0 flush table #10263: started
>2018-07-11 05:54:33.456539 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473456525, "cf_name": "default", "job": 757, "event": "table_file_creation", "file_number": 10263, "file_size": 2151539, "table_properties": {"data_size": 2134707, "index_size": 8657, "filter_size": 7251, "raw_key_size": 11604, "raw_average_key_size": 20, "raw_value_size": 2120596, "raw_average_value_size": 3746, "num_data_blocks": 263, "num_entries": 566, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:54:33.456550 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 757] Level-0 flush table #10263: 2151539 bytes OK
>2018-07-11 05:54:33.514892 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.473083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10263 started
>2018-07-11 05:54:33.514900 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.514855) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10263: memtable #1 done
>2018-07-11 05:54:33.514902 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.514870) EVENT_LOG_v1 {"time_micros": 1531288473514863, "job": 757, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:54:33.514904 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.514881) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:54:33.514923 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 757] Try to delete WAL files size 2205212, prev total WAL file size 2205212, number of live WAL files 2.
>
>2018-07-11 05:54:33.515521 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:54:33.515584 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.515577) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:54:33.515594 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 758] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:54:33.515600 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 757 Base level 0, inputs: [10263(2101KB)], [10261(13MB)]
>
>2018-07-11 05:54:33.515613 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473515603, "job": 758, "event": "compaction_started", "files_L0": [10263], "files_L1": [10261], "score": -1, "input_data_size": 16526472}
>2018-07-11 05:54:33.690610 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 758] Generated table #10264: 3127 keys, 14528234 bytes
>2018-07-11 05:54:33.690658 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473690630, "cf_name": "default", "job": 758, "event": "table_file_creation", "file_number": 10264, "file_size": 14528234, "table_properties": {"data_size": 14416803, "index_size": 63458, "filter_size": 47046, "raw_key_size": 63045, "raw_average_key_size": 20, "raw_value_size": 14327695, "raw_average_value_size": 4581, "num_data_blocks": 1852, "num_entries": 3127, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:54:33.723789 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 758] Compacted 1@0 + 1@1 files to L1 => 14528234 bytes
>2018-07-11 05:54:33.765659 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.765592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.4 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(2.1, 13.7) out(13.9), read-write-amplify(14.4) write-amplify(6.8) OK, records in: 3641, records dropped: 514
>
>2018-07-11 05:54:33.765668 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:54:33.765613) EVENT_LOG_v1 {"time_micros": 1531288473765604, "job": 758, "event": "compaction_finished", "compaction_time_micros": 208160, "output_level": 1, "num_output_files": 1, "total_output_size": 14528234, "num_input_records": 3641, "num_output_records": 3127, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:54:33.766198 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473766193, "job": 758, "event": "table_file_deletion", "file_number": 10263}
>2018-07-11 05:54:33.768844 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288473768841, "job": 758, "event": "table_file_deletion", "file_number": 10261}
>2018-07-11 05:54:35.496582 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 05:54:35.496609 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1336508081' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 05:54:36.394936 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["2"]} v 0) v1
>2018-07-11 05:54:36.394975 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.2 10.8.128.106:6800/12651' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["2"]}]: dispatch
>2018-07-11 05:54:36.395700 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 2, "weight":0.9096, "args": ["host=magna106", "root=default"]} v 0) v1
>2018-07-11 05:54:36.395738 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.2 10.8.128.106:6800/12651' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.9096, "args": ["host=magna106", "root=default"]}]: dispatch
>2018-07-11 05:54:36.395831 7f29191ed700  0 mon.magna102@0(leader).osd e464 create-or-move crush item name 'osd.2' initial_weight 0.9096 at location {host=magna106,root=default}
>2018-07-11 05:54:36.525058 7f29149e4700  1 mon.magna102@0(leader).log v19392 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19392)
>2018-07-11 05:54:37.559012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 osds down (OSD_DOWN)
>2018-07-11 05:54:37.559047 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 hosts (6 osds) down (OSD_HOST_DOWN)
>2018-07-11 05:54:37.574151 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]} v 0) v1
>2018-07-11 05:54:37.574186 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.7 10.8.128.114:6804/15885' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["7"]}]: dispatch
>2018-07-11 05:54:37.658750 7f29149e4700  1 mon.magna102@0(leader).osd e465 e465: 12 total, 4 up, 12 in
>2018-07-11 05:54:37.692274 7f29149e4700  0 log_channel(cluster) log [INF] : osd.2 10.8.128.106:6800/12651 boot
>2018-07-11 05:54:37.692518 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e465: 12 total, 4 up, 12 in
>2018-07-11 05:54:37.725617 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
>2018-07-11 05:54:37.725644 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
>2018-07-11 05:54:37.725888 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight":0.9096, "args": ["host=magna114", "root=default"]} v 0) v1
>2018-07-11 05:54:37.725923 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.7 10.8.128.114:6804/15885' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.9096, "args": ["host=magna114", "root=default"]}]: dispatch
>2018-07-11 05:54:37.726001 7f29191ed700  0 mon.magna102@0(leader).osd e465 create-or-move crush item name 'osd.7' initial_weight 0.9096 at location {host=magna114,root=default}
>2018-07-11 05:54:37.759137 7f29149e4700  1 mon.magna102@0(leader).log v19393 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 19393)
>2018-07-11 05:54:37.842683 7f29149e4700  1 mon.magna102@0(leader).osd e466 e466: 12 total, 4 up, 12 in
>2018-07-11 05:54:37.876278 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e466: 12 total, 4 up, 12 in
>2018-07-11 05:54:38.063386 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["5"]} v 0) v1
>2018-07-11 05:54:38.063416 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.5 10.8.128.113:6804/15263' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["5"]}]: dispatch
>2018-07-11 05:54:38.063965 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 5, "weight":0.9096, "args": ["host=magna113", "root=default"]} v 0) v1
>2018-07-11 05:54:38.064001 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.5 10.8.128.113:6804/15263' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.9096, "args": ["host=magna113", "root=default"]}]: dispatch
>2018-07-11 05:54:38.064090 7f29191ed700  0 mon.magna102@0(leader).osd e466 create-or-move crush item name 'osd.5' initial_weight 0.9096 at location {host=magna113,root=default}
>2018-07-11 05:54:38.826367 7f29149e4700  1 mon.magna102@0(leader).log v19394 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 19394)
>2018-07-11 05:54:38.909932 7f29149e4700  1 mon.magna102@0(leader).osd e467 e467: 12 total, 4 up, 12 in
>2018-07-11 05:54:38.943504 7f29149e4700  0 log_channel(cluster) log [INF] : osd.5 10.8.128.113:6804/15263 boot
>2018-07-11 05:54:38.943579 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e467: 12 total, 4 up, 12 in
>2018-07-11 05:54:38.943722 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 5} v 0) v1
>2018-07-11 05:54:38.943748 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch
>2018-07-11 05:54:40.043738 7f29149e4700  1 mon.magna102@0(leader).osd e468 e468: 12 total, 5 up, 12 in
>2018-07-11 05:54:40.077163 7f29149e4700  1 mon.magna102@0(leader).log v19395 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19395)
>2018-07-11 05:54:40.077407 7f29149e4700  0 log_channel(cluster) log [INF] : osd.7 10.8.128.114:6804/15885 boot
>2018-07-11 05:54:40.077546 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e468: 12 total, 5 up, 12 in
>2018-07-11 05:54:40.077666 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 7} v 0) v1
>2018-07-11 05:54:40.077697 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch
>2018-07-11 05:54:40.227378 7f29149e4700  1 mon.magna102@0(leader).osd e469 e469: 12 total, 5 up, 12 in
>2018-07-11 05:54:40.261036 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e469: 12 total, 5 up, 12 in
>2018-07-11 05:54:41.110999 7f29149e4700  1 mon.magna102@0(leader).log v19396 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19396)
>2018-07-11 05:54:43.423418 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217440 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:54:43.423458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 7 pgs inactive, 14 pgs down, 35 pgs peering (PG_AVAILABILITY)
>2018-07-11 05:54:43.423508 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2466594/24217440 objects degraded (10.185%), 70 pgs degraded (PG_DEGRADED)
>2018-07-11 05:54:43.578214 7f29149e4700  1 mon.magna102@0(leader).log v19397 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 19397)
>2018-07-11 05:54:45.461502 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 7 osds down (OSD_DOWN)
>2018-07-11 05:54:45.461537 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 hosts (3 osds) down (OSD_HOST_DOWN)
>2018-07-11 05:54:45.687006 7f29149e4700  1 mon.magna102@0(leader).log v19398 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19398)
>2018-07-11 05:54:48.389685 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["0"]} v 0) v1
>2018-07-11 05:54:48.389715 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.0 10.8.128.113:6800/15255' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["0"]}]: dispatch
>2018-07-11 05:54:48.390213 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 0, "weight":0.9096, "args": ["host=magna113", "root=default"]} v 0) v1
>2018-07-11 05:54:48.390247 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.0 10.8.128.113:6800/15255' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.9096, "args": ["host=magna113", "root=default"]}]: dispatch
>2018-07-11 05:54:48.390332 7f29191ed700  0 mon.magna102@0(leader).osd e469 create-or-move crush item name 'osd.0' initial_weight 0.9096 at location {host=magna113,root=default}
>2018-07-11 05:54:48.562323 7f29149e4700  1 mon.magna102@0(leader).log v19399 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19399)
>2018-07-11 05:54:48.948242 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]} v 0) v1
>2018-07-11 05:54:48.948273 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.1 10.8.128.106:6804/12677' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]}]: dispatch
>2018-07-11 05:54:48.948741 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 1, "weight":0.9096, "args": ["host=magna106", "root=default"]} v 0) v1
>2018-07-11 05:54:48.948775 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.1 10.8.128.106:6804/12677' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.9096, "args": ["host=magna106", "root=default"]}]: dispatch
>2018-07-11 05:54:48.948859 7f29191ed700  0 mon.magna102@0(leader).osd e469 create-or-move crush item name 'osd.1' initial_weight 0.9096 at location {host=magna106,root=default}
>2018-07-11 05:54:49.605294 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 7 pgs inactive, 14 pgs down, 36 pgs peering (PG_AVAILABILITY)
>2018-07-11 05:54:49.647190 7f29149e4700  1 mon.magna102@0(leader).osd e470 e470: 12 total, 7 up, 12 in
>2018-07-11 05:54:49.689045 7f29149e4700  0 log_channel(cluster) log [INF] : osd.0 10.8.128.113:6800/15255 boot
>2018-07-11 05:54:49.689130 7f29149e4700  0 log_channel(cluster) log [INF] : osd.1 10.8.128.106:6804/12677 boot
>2018-07-11 05:54:49.689206 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e470: 12 total, 7 up, 12 in
>2018-07-11 05:54:49.730476 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1
>2018-07-11 05:54:49.730502 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch
>2018-07-11 05:54:49.730772 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 1} v 0) v1
>2018-07-11 05:54:49.730796 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch
>2018-07-11 05:54:49.772544 7f29149e4700  1 mon.magna102@0(leader).log v19400 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19400)
>2018-07-11 05:54:49.829820 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["8"]} v 0) v1
>2018-07-11 05:54:49.829849 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.8 10.8.128.108:6800/11967' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["8"]}]: dispatch
>2018-07-11 05:54:49.864310 7f29149e4700  1 mon.magna102@0(leader).osd e471 e471: 12 total, 7 up, 12 in
>2018-07-11 05:54:49.906582 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e471: 12 total, 7 up, 12 in
>2018-07-11 05:54:49.907097 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 8, "weight":0.9096, "args": ["host=magna108", "root=default"]} v 0) v1
>2018-07-11 05:54:49.907133 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.8 10.8.128.108:6800/11967' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.9096, "args": ["host=magna108", "root=default"]}]: dispatch
>2018-07-11 05:54:49.907232 7f29191ed700  0 mon.magna102@0(leader).osd e471 create-or-move crush item name 'osd.8' initial_weight 0.9096 at location {host=magna108,root=default}
>2018-07-11 05:54:50.461769 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 5 osds down (OSD_DOWN)
>2018-07-11 05:54:50.865062 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 host (2 osds) down (OSD_HOST_DOWN)
>2018-07-11 05:54:50.889681 7f29149e4700  1 mon.magna102@0(leader).log v19401 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 6 entries (version 19401)
>2018-07-11 05:54:50.973197 7f29149e4700  1 mon.magna102@0(leader).osd e472 e472: 12 total, 8 up, 12 in
>2018-07-11 05:54:51.015047 7f29149e4700  0 log_channel(cluster) log [INF] : osd.8 10.8.128.108:6800/11967 boot
>2018-07-11 05:54:51.015155 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e472: 12 total, 8 up, 12 in
>2018-07-11 05:54:51.015322 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 8} v 0) v1
>2018-07-11 05:54:51.015348 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch
>2018-07-11 05:54:51.165241 7f29149e4700  1 mon.magna102@0(leader).osd e473 e473: 12 total, 8 up, 12 in
>2018-07-11 05:54:51.207322 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e473: 12 total, 8 up, 12 in
>2018-07-11 05:54:51.210901 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["9"]} v 0) v1
>2018-07-11 05:54:51.210932 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.9 10.8.128.107:6804/12617' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["9"]}]: dispatch
>2018-07-11 05:54:51.211608 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 9, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 05:54:51.211645 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.9 10.8.128.107:6804/12617' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 05:54:51.211738 7f29191ed700  0 mon.magna102@0(leader).osd e473 create-or-move crush item name 'osd.9' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 05:54:52.007262 7f29149e4700  1 mon.magna102@0(leader).log v19402 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 19402)
>2018-07-11 05:54:52.165925 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_HOST_DOWN (was: 1 host (2 osds) down)
>2018-07-11 05:54:52.249164 7f29149e4700  1 mon.magna102@0(leader).osd e474 e474: 12 total, 9 up, 12 in
>2018-07-11 05:54:52.291016 7f29149e4700  0 log_channel(cluster) log [INF] : osd.9 10.8.128.107:6804/12617 boot
>2018-07-11 05:54:52.291215 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e474: 12 total, 9 up, 12 in
>2018-07-11 05:54:52.332534 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 9} v 0) v1
>2018-07-11 05:54:52.332560 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch
>2018-07-11 05:54:52.466326 7f29149e4700  1 mon.magna102@0(leader).osd e475 e475: 12 total, 9 up, 12 in
>2018-07-11 05:54:52.508425 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e475: 12 total, 9 up, 12 in
>2018-07-11 05:54:53.091556 7f29149e4700  1 mon.magna102@0(leader).log v19403 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19403)
>2018-07-11 05:54:53.550167 7f29149e4700  1 mon.magna102@0(leader).osd e476 e476: 12 total, 9 up, 12 in
>2018-07-11 05:54:53.592143 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e476: 12 total, 9 up, 12 in
>2018-07-11 05:54:54.091538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2098674/24217440 objects degraded (8.666%), 62 pgs degraded (PG_DEGRADED)
>2018-07-11 05:54:54.182896 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["11"]} v 0) v1
>2018-07-11 05:54:54.182926 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.11 10.8.128.114:6800/15919' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["11"]}]: dispatch
>2018-07-11 05:54:54.183519 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 11, "weight":0.9096, "args": ["host=magna114", "root=default"]} v 0) v1
>2018-07-11 05:54:54.183553 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.11 10.8.128.114:6800/15919' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.9096, "args": ["host=magna114", "root=default"]}]: dispatch
>2018-07-11 05:54:54.183642 7f29191ed700  0 mon.magna102@0(leader).osd e476 create-or-move crush item name 'osd.11' initial_weight 0.9096 at location {host=magna114,root=default}
>2018-07-11 05:54:54.634023 7f29149e4700  1 mon.magna102@0(leader).log v19404 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 19404)
>2018-07-11 05:54:55.267630 7f29149e4700  1 mon.magna102@0(leader).osd e477 e477: 12 total, 8 up, 12 in
>2018-07-11 05:54:55.309530 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e477: 12 total, 8 up, 12 in
>2018-07-11 05:54:55.462028 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 osds down (OSD_DOWN)
>2018-07-11 05:54:55.462077 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 6 pgs inactive, 1 pg down, 49 pgs peering (PG_AVAILABILITY)
>2018-07-11 05:54:56.451458 7f29149e4700  1 mon.magna102@0(leader).osd e478 e478: 12 total, 9 up, 12 in
>2018-07-11 05:54:56.493164 7f29149e4700  1 mon.magna102@0(leader).log v19405 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19405)
>2018-07-11 05:54:56.493495 7f29149e4700  0 log_channel(cluster) log [INF] : osd.11 10.8.128.114:6800/15919 boot
>2018-07-11 05:54:56.493833 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e478: 12 total, 9 up, 12 in
>2018-07-11 05:54:56.494157 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 11} v 0) v1
>2018-07-11 05:54:56.494217 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch
>2018-07-11 05:54:56.635025 7f29149e4700  1 mon.magna102@0(leader).osd e479 e479: 12 total, 9 up, 12 in
>2018-07-11 05:54:56.677032 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e479: 12 total, 9 up, 12 in
>2018-07-11 05:54:57.543656 7f29149e4700  1 mon.magna102@0(leader).log v19406 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19406)
>2018-07-11 05:54:59.431639 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2473926/24217440 objects degraded (10.215%), 71 pgs degraded (PG_DEGRADED)
>2018-07-11 05:54:59.602518 7f29149e4700  1 mon.magna102@0(leader).log v19407 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19407)
>2018-07-11 05:55:00.350888 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:55:00.462312 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 osds down (OSD_DOWN)
>2018-07-11 05:55:00.462360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 76 pgs peering, 18 pgs stale (PG_AVAILABILITY)
>2018-07-11 05:55:00.669261 7f29149e4700  1 mon.magna102@0(leader).log v19408 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19408)
>2018-07-11 05:55:01.393012 7f29191ed700  1 mon.magna102@0(leader).osd e479 prepare_failure osd.6 10.8.128.115:6800/2787 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 05:55:01.393029 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.6 10.8.128.115:6800/2787 reported failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 05:55:01.669298 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217440 objects misplaced (0.011%))
>2018-07-11 05:55:01.803029 7f29149e4700  1 mon.magna102@0(leader).log v19409 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19409)
>2018-07-11 05:55:02.061728 7f29191ed700  1 mon.magna102@0(leader).osd e479 prepare_failure osd.6 10.8.128.115:6800/2787 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 05:55:02.061742 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.6 10.8.128.115:6800/2787 reported failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 05:55:02.572676 7f29191ed700  1 mon.magna102@0(leader).osd e479 prepare_failure osd.6 10.8.128.115:6800/2787 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 05:55:02.572691 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.6 10.8.128.115:6800/2787 reported failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 05:55:02.710160 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:02.710193 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/981566363' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:02.869980 7f29149e4700  1 mon.magna102@0(leader).log v19410 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19410)
>2018-07-11 05:55:03.807254 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["6"]} v 0) v1
>2018-07-11 05:55:03.807292 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.6 10.8.128.115:6800/13323' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["6"]}]: dispatch
>2018-07-11 05:55:03.807894 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 6, "weight":0.9096, "args": ["host=magna115", "root=default"]} v 0) v1
>2018-07-11 05:55:03.807930 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.6 10.8.128.115:6800/13323' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.9096, "args": ["host=magna115", "root=default"]}]: dispatch
>2018-07-11 05:55:03.808021 7f29191ed700  0 mon.magna102@0(leader).osd e479 create-or-move crush item name 'osd.6' initial_weight 0.9096 at location {host=magna115,root=default}
>2018-07-11 05:55:03.936632 7f29149e4700  1 mon.magna102@0(leader).log v19411 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19411)
>2018-07-11 05:55:04.699918 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:04.699941 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/620528863' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:04.937568 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 host (1 osds) down (OSD_HOST_DOWN)
>2018-07-11 05:55:05.003983 7f29149e4700  1 mon.magna102@0(leader).osd e480 e480: 12 total, 8 up, 12 in
>2018-07-11 05:55:05.037680 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e480: 12 total, 8 up, 12 in
>2018-07-11 05:55:05.104174 7f29149e4700  1 mon.magna102@0(leader).log v19412 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19412)
>2018-07-11 05:55:05.462572 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 osds down (OSD_DOWN)
>2018-07-11 05:55:05.462621 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg down, 67 pgs peering, 10 pgs stale (PG_AVAILABILITY)
>2018-07-11 05:55:05.462643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2948171/24217440 objects degraded (12.174%), 85 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:06.004601 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_HOST_DOWN (was: 1 host (1 osds) down)
>2018-07-11 05:55:06.079519 7f29149e4700  1 mon.magna102@0(leader).osd e481 e481: 12 total, 9 up, 12 in
>2018-07-11 05:55:06.114273 7f29149e4700  0 log_channel(cluster) log [INF] : osd.6 10.8.128.115:6800/13323 boot
>2018-07-11 05:55:06.114747 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e481: 12 total, 9 up, 12 in
>2018-07-11 05:55:06.114828 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:06.114862 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4242066206' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:06.147476 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 6} v 0) v1
>2018-07-11 05:55:06.147507 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch
>2018-07-11 05:55:06.247968 7f29149e4700  1 mon.magna102@0(leader).log v19413 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 19413)
>2018-07-11 05:55:06.314741 7f29149e4700  1 mon.magna102@0(leader).osd e482 e482: 12 total, 9 up, 12 in
>2018-07-11 05:55:06.348708 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e482: 12 total, 9 up, 12 in
>2018-07-11 05:55:07.315200 7f29149e4700  1 mon.magna102@0(leader).log v19414 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19414)
>2018-07-11 05:55:07.381908 7f29149e4700  1 mon.magna102@0(leader).osd e483 e483: 12 total, 9 up, 12 in
>2018-07-11 05:55:07.415559 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e483: 12 total, 9 up, 12 in
>2018-07-11 05:55:07.481022 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:07.481045 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/920689136' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:08.515779 7f29149e4700  1 mon.magna102@0(leader).log v19415 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19415)
>2018-07-11 05:55:08.881580 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:08.881603 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2905046533' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:09.649448 7f29149e4700  1 mon.magna102@0(leader).log v19416 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19416)
>2018-07-11 05:55:10.226905 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:10.226928 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/272456586' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:10.462901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 osds down (OSD_DOWN)
>2018-07-11 05:55:10.462968 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg down, 18 pgs peering, 7 pgs incomplete, 2 pgs stale (PG_AVAILABILITY)
>2018-07-11 05:55:10.462999 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 3986292/24217440 objects degraded (16.460%), 118 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:10.724847 7f29149e4700  1 mon.magna102@0(leader).log v19417 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 19417)
>2018-07-11 05:55:11.545630 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:11.545652 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/828039719' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:11.791592 7f29149e4700  1 mon.magna102@0(leader).log v19418 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19418)
>2018-07-11 05:55:12.975473 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:12.975496 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3297974825' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:13.091986 7f29149e4700  1 mon.magna102@0(leader).log v19419 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19419)
>2018-07-11 05:55:14.300907 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:14.300930 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4005113851' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:15.225545 7f29149e4700  1 mon.magna102@0(leader).log v19420 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19420)
>2018-07-11 05:55:15.463245 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg down, 9 pgs incomplete, 2 pgs stale (PG_AVAILABILITY)
>2018-07-11 05:55:15.463291 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 4062480/24217440 objects degraded (16.775%), 122 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:15.652217 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:15.652241 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1009327809' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:16.292581 7f29149e4700  1 mon.magna102@0(leader).log v19421 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19421)
>2018-07-11 05:55:17.142105 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:17.142128 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/999777345' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:17.426287 7f29149e4700  1 mon.magna102@0(leader).log v19422 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19422)
>2018-07-11 05:55:17.902165 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["3"]} v 0) v1
>2018-07-11 05:55:17.902210 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.3 10.8.128.106:6801/12650' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["3"]}]: dispatch
>2018-07-11 05:55:17.902806 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 3, "weight":0.9096, "args": ["host=magna106", "root=default"]} v 0) v1
>2018-07-11 05:55:17.902841 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.3 10.8.128.106:6801/12650' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.9096, "args": ["host=magna106", "root=default"]}]: dispatch
>2018-07-11 05:55:17.902926 7f29191ed700  0 mon.magna102@0(leader).osd e483 create-or-move crush item name 'osd.3' initial_weight 0.9096 at location {host=magna106,root=default}
>2018-07-11 05:55:18.510607 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 osds down (OSD_DOWN)
>2018-07-11 05:55:18.576758 7f29149e4700  1 mon.magna102@0(leader).osd e484 e484: 12 total, 10 up, 12 in
>2018-07-11 05:55:18.610247 7f29149e4700  0 log_channel(cluster) log [INF] : osd.3 10.8.128.106:6801/12650 boot
>2018-07-11 05:55:18.610435 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e484: 12 total, 10 up, 12 in
>2018-07-11 05:55:18.643418 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
>2018-07-11 05:55:18.643451 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
>2018-07-11 05:55:18.647694 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:18.647712 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/664744156' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:18.677067 7f29149e4700  1 mon.magna102@0(leader).log v19423 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19423)
>2018-07-11 05:55:18.768971 7f29149e4700  1 mon.magna102@0(leader).osd e485 e485: 12 total, 10 up, 12 in
>2018-07-11 05:55:18.802848 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e485: 12 total, 10 up, 12 in
>2018-07-11 05:55:19.744223 7f29149e4700  1 mon.magna102@0(leader).log v19424 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19424)
>2018-07-11 05:55:19.836087 7f29149e4700  1 mon.magna102@0(leader).osd e486 e486: 12 total, 10 up, 12 in
>2018-07-11 05:55:19.869763 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e486: 12 total, 10 up, 12 in
>2018-07-11 05:55:20.111141 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:20.111164 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1458328775' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:20.463557 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 4062480/24217443 objects degraded (16.775%), 122 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:20.911783 7f29149e4700  1 mon.magna102@0(leader).log v19425 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19425)
>2018-07-11 05:55:21.520309 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:21.520333 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/410679290' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:21.978533 7f29149e4700  1 mon.magna102@0(leader).log v19426 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19426)
>2018-07-11 05:55:22.975148 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:22.975172 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1567361579' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:23.112314 7f29149e4700  1 mon.magna102@0(leader).log v19427 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19427)
>2018-07-11 05:55:24.112329 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg peering, 5 pgs incomplete (PG_AVAILABILITY)
>2018-07-11 05:55:24.262653 7f29149e4700  1 mon.magna102@0(leader).log v19428 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19428)
>2018-07-11 05:55:24.799712 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:24.799736 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/327630081' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:25.329496 7f29149e4700  1 mon.magna102@0(leader).log v19429 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19429)
>2018-07-11 05:55:25.463872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 3356232/24217446 objects degraded (13.859%), 103 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:26.396324 7f29149e4700  1 mon.magna102@0(leader).log v19430 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19430)
>2018-07-11 05:55:26.547729 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:26.547752 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2459223494' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:27.598119 7f29149e4700  1 mon.magna102@0(leader).log v19431 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19431)
>2018-07-11 05:55:28.362165 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:28.362199 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3982370755' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:28.665099 7f29149e4700  1 mon.magna102@0(leader).log v19432 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19432)
>2018-07-11 05:55:29.493833 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["4"]} v 0) v1
>2018-07-11 05:55:29.493873 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.4 10.8.128.113:6808/15287' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["4"]}]: dispatch
>2018-07-11 05:55:29.494600 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 4, "weight":0.9096, "args": ["host=magna113", "root=default"]} v 0) v1
>2018-07-11 05:55:29.494638 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.4 10.8.128.113:6808/15287' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.9096, "args": ["host=magna113", "root=default"]}]: dispatch
>2018-07-11 05:55:29.494732 7f29191ed700  0 mon.magna102@0(leader).osd e486 create-or-move crush item name 'osd.4' initial_weight 0.9096 at location {host=magna113,root=default}
>2018-07-11 05:55:29.706811 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg peering, 4 pgs incomplete (PG_AVAILABILITY)
>2018-07-11 05:55:29.740379 7f29149e4700  1 mon.magna102@0(leader).log v19433 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19433)
>2018-07-11 05:55:29.782695 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 osds down (OSD_DOWN)
>2018-07-11 05:55:29.883457 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:29.883478 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/86189454' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:29.890653 7f29149e4700  1 mon.magna102@0(leader).osd e487 e487: 12 total, 11 up, 12 in
>2018-07-11 05:55:29.924349 7f29149e4700  0 log_channel(cluster) log [INF] : osd.4 10.8.128.113:6808/15287 boot
>2018-07-11 05:55:29.924557 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e487: 12 total, 11 up, 12 in
>2018-07-11 05:55:29.924747 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 4} v 0) v1
>2018-07-11 05:55:29.924832 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch
>2018-07-11 05:55:30.091024 7f29149e4700  1 mon.magna102@0(leader).osd e488 e488: 12 total, 11 up, 12 in
>2018-07-11 05:55:30.124831 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e488: 12 total, 11 up, 12 in
>2018-07-11 05:55:30.464152 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 3363979/24217446 objects degraded (13.891%), 99 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:30.808053 7f29149e4700  1 mon.magna102@0(leader).log v19434 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 19434)
>2018-07-11 05:55:31.174825 7f29149e4700  1 mon.magna102@0(leader).osd e489 e489: 12 total, 11 up, 12 in
>2018-07-11 05:55:31.216957 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e489: 12 total, 11 up, 12 in
>2018-07-11 05:55:31.314508 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:31.314530 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3910543899' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:32.258455 7f29149e4700  1 mon.magna102@0(leader).log v19435 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19435)
>2018-07-11 05:55:32.342069 7f29149e4700  1 mon.magna102@0(leader).osd e490 e490: 12 total, 11 up, 12 in
>2018-07-11 05:55:32.384122 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e490: 12 total, 11 up, 12 in
>2018-07-11 05:55:32.584079 7f29149e4700  1 mon.magna102@0(leader).osd e491 e491: 12 total, 11 up, 12 in
>2018-07-11 05:55:32.626039 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e491: 12 total, 11 up, 12 in
>2018-07-11 05:55:32.928767 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:32.928790 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3907713617' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:33.425950 7f29149e4700  1 mon.magna102@0(leader).log v19436 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19436)
>2018-07-11 05:55:34.495067 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:34.495091 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1212773527' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:34.751405 7f29149e4700  1 mon.magna102@0(leader).log v19437 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19437)
>2018-07-11 05:55:35.464490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs incomplete (PG_AVAILABILITY)
>2018-07-11 05:55:35.464537 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2718118/24217449 objects degraded (11.224%), 83 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:35.607320 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 05:55:35.607358 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/12665' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 05:55:35.607890 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 05:55:35.607925 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/12665' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 05:55:35.608026 7f29191ed700  0 mon.magna102@0(leader).osd e491 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 05:55:35.793118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217449 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:55:35.793158 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs incomplete)
>2018-07-11 05:55:35.834897 7f29149e4700  1 mon.magna102@0(leader).log v19438 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 19438)
>2018-07-11 05:55:36.119439 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:36.119462 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2649149123' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:36.877676 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 05:55:36.918729 7f29149e4700  1 mon.magna102@0(leader).log v19439 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19439)
>2018-07-11 05:55:37.002345 7f29149e4700  1 mon.magna102@0(leader).osd e492 e492: 12 total, 12 up, 12 in
>2018-07-11 05:55:37.044279 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/12665 boot
>2018-07-11 05:55:37.044499 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e492: 12 total, 12 up, 12 in
>2018-07-11 05:55:37.044655 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 05:55:37.044690 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 05:55:37.227601 7f29149e4700  1 mon.magna102@0(leader).osd e493 e493: 12 total, 12 up, 12 in
>2018-07-11 05:55:37.269894 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e493: 12 total, 12 up, 12 in
>2018-07-11 05:55:37.541127 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:37.541151 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3939858542' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:38.002964 7f29149e4700  1 mon.magna102@0(leader).log v19440 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19440)
>2018-07-11 05:55:38.311449 7f29149e4700  1 mon.magna102@0(leader).osd e494 e494: 12 total, 12 up, 12 in
>2018-07-11 05:55:38.345163 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e494: 12 total, 12 up, 12 in
>2018-07-11 05:55:38.850901 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:38.850925 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/537907644' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:39.378646 7f29149e4700  1 mon.magna102@0(leader).log v19441 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19441)
>2018-07-11 05:55:40.330602 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:40.330624 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/627115929' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:40.478796 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1238023/24217449 objects degraded (5.112%), 38 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:40.512311 7f29149e4700  1 mon.magna102@0(leader).log v19442 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19442)
>2018-07-11 05:55:41.637792 7f29149e4700  1 mon.magna102@0(leader).log v19443 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19443)
>2018-07-11 05:55:43.457478 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217452 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:55:43.604781 7f29149e4700  1 mon.magna102@0(leader).log v19444 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19444)
>2018-07-11 05:55:43.780241 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:43.780267 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1349396109' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:43.781615 7f29191ed700  1 mon.magna102@0(leader).log v19444 check_sub sending message to client.? 10.8.128.102:0/1349396109 with 1 entries (version 19444)
>2018-07-11 05:55:44.671619 7f29149e4700  1 mon.magna102@0(leader).log v19445 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19445)
>2018-07-11 05:55:44.671653 7f29149e4700  1 mon.magna102@0(leader).log v19445 check_sub sending message to client.? 10.8.128.102:0/1349396109 with 0 entries (version 19445)
>2018-07-11 05:55:45.479075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 498/24217452 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED)
>2018-07-11 05:55:45.738562 7f29149e4700  1 mon.magna102@0(leader).log v19446 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19446)
>2018-07-11 05:55:45.738608 7f29149e4700  1 mon.magna102@0(leader).log v19446 check_sub sending message to client.? 10.8.128.102:0/1349396109 with 1 entries (version 19446)
>2018-07-11 05:55:50.049558 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:50.049581 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2702628610' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:50.479353 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217455 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:55:50.479406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 497/24217455 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:55:50.598871 7f29149e4700  1 mon.magna102@0(leader).log v19447 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19447)
>2018-07-11 05:55:52.330758 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:52.330781 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1564070779' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:52.740820 7f29149e4700  1 mon.magna102@0(leader).log v19448 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19448)
>2018-07-11 05:55:53.998287 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:53.998310 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1299901596' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:54.874503 7f29149e4700  1 mon.magna102@0(leader).log v19449 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19449)
>2018-07-11 05:55:55.780552 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:55.780575 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2704363047' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:55.916267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 496/24217455 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:55:55.949783 7f29149e4700  1 mon.magna102@0(leader).log v19450 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19450)
>2018-07-11 05:55:57.016852 7f29149e4700  1 mon.magna102@0(leader).log v19451 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19451)
>2018-07-11 05:55:57.509132 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:55:57.509156 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2640997500' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:55:58.050284 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217458 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:55:58.083694 7f29149e4700  1 mon.magna102@0(leader).log v19452 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19452)
>2018-07-11 05:55:59.159075 7f29149e4700  1 mon.magna102@0(leader).log v19453 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19453)
>2018-07-11 05:56:00.351067 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:56:02.223116 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:02.223140 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2155093879' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:02.224446 7f29191ed700  1 mon.magna102@0(leader).log v19453 check_sub sending message to client.? 10.8.128.102:0/2155093879 with 1 entries (version 19453)
>2018-07-11 05:56:02.617903 7f29149e4700  1 mon.magna102@0(leader).log v19454 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19454)
>2018-07-11 05:56:02.617936 7f29149e4700  1 mon.magna102@0(leader).log v19454 check_sub sending message to client.? 10.8.128.102:0/2155093879 with 0 entries (version 19454)
>2018-07-11 05:56:05.468597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217461 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:05.468642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 496/24217461 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:05.618457 7f29149e4700  1 mon.magna102@0(leader).log v19455 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19455)
>2018-07-11 05:56:05.618492 7f29149e4700  1 mon.magna102@0(leader).log v19455 check_sub sending message to client.? 10.8.128.102:0/2155093879 with 2 entries (version 19455)
>2018-07-11 05:56:10.501626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217464 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:10.501657 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 495/24217464 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:10.610544 7f29149e4700  1 mon.magna102@0(leader).log v19456 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19456)
>2018-07-11 05:56:10.610591 7f29149e4700  1 mon.magna102@0(leader).log v19456 check_sub sending message to client.? 10.8.128.102:0/2155093879 with 2 entries (version 19456)
>2018-07-11 05:56:19.475153 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217467 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:19.475212 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 495/24217467 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:19.628215 7f29149e4700  1 mon.magna102@0(leader).log v19457 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19457)
>2018-07-11 05:56:22.783140 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:22.783163 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4062135061' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:22.903595 7f29149e4700  1 mon.magna102@0(leader).log v19458 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19458)
>2018-07-11 05:56:24.452683 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:24.452706 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3091891293' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:25.045604 7f29149e4700  1 mon.magna102@0(leader).log v19459 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19459)
>2018-07-11 05:56:26.045622 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217470 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:26.045675 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 494/24217470 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:26.079132 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:26.079153 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1728724673' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:26.196021 7f29149e4700  1 mon.magna102@0(leader).log v19460 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19460)
>2018-07-11 05:56:32.442365 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:32.442388 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3084945680' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:32.647911 7f29149e4700  1 mon.magna102@0(leader).log v19461 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19461)
>2018-07-11 05:56:35.339866 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:35.339888 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/559729841' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:35.473356 7f29149e4700  1 mon.magna102@0(leader).log v19462 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19462)
>2018-07-11 05:56:35.483162 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217473 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:35.483218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 494/24217473 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:36.557004 7f29149e4700  1 mon.magna102@0(leader).log v19463 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19463)
>2018-07-11 05:56:37.658746 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:37.658769 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1197572701' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:38.723984 7f29149e4700  1 mon.magna102@0(leader).log v19464 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19464)
>2018-07-11 05:56:40.195021 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:40.195044 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1944480593' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:40.523667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 493/24217473 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:40.891078 7f29149e4700  1 mon.magna102@0(leader).log v19465 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19465)
>2018-07-11 05:56:41.891041 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217476 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:41.955317 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:41.955339 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2228200752' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:42.066432 7f29149e4700  1 mon.magna102@0(leader).log v19466 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19466)
>2018-07-11 05:56:43.561828 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:56:43.561852 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3293668049' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:56:43.683417 7f29149e4700  1 mon.magna102@0(leader).log v19467 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19467)
>2018-07-11 05:56:45.525111 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 493/24217476 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:45.642114 7f29149e4700  1 mon.magna102@0(leader).log v19468 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19468)
>2018-07-11 05:56:49.490329 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217479 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:49.650924 7f29149e4700  1 mon.magna102@0(leader).log v19469 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19469)
>2018-07-11 05:56:50.525385 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 493/24217479 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:50.717683 7f29149e4700  1 mon.magna102@0(leader).log v19470 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19470)
>2018-07-11 05:56:50.901911 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd lspools"} v 0) v1
>2018-07-11 05:56:50.901938 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3780143427' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch
>2018-07-11 05:56:51.851520 7f29149e4700  1 mon.magna102@0(leader).log v19471 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19471)
>2018-07-11 05:56:55.493244 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217482 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:56:55.643635 7f29149e4700  1 mon.magna102@0(leader).log v19472 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19472)
>2018-07-11 05:56:59.494983 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 492/24217485 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:56:59.644144 7f29149e4700  1 mon.magna102@0(leader).log v19473 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19473)
>2018-07-11 05:57:00.351244 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:57:00.526895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217485 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:00.710983 7f29149e4700  1 mon.magna102@0(leader).log v19474 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19474)
>2018-07-11 05:57:09.499633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217488 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:09.499682 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 492/24217488 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:09.653677 7f29149e4700  1 mon.magna102@0(leader).log v19475 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19475)
>2018-07-11 05:57:10.862131 7f29149e4700  1 mon.magna102@0(leader).log v19476 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19476)
>2018-07-11 05:57:15.502384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217491 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:15.502430 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 491/24217491 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:15.655630 7f29149e4700  1 mon.magna102@0(leader).log v19477 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19477)
>2018-07-11 05:57:16.722513 7f29149e4700  1 mon.magna102@0(leader).log v19478 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19478)
>2018-07-11 05:57:23.505663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217494 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:23.505696 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 491/24217494 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:23.656677 7f29149e4700  1 mon.magna102@0(leader).log v19479 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19479)
>2018-07-11 05:57:29.508509 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217497 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:29.508555 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 490/24217497 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:29.674010 7f29149e4700  1 mon.magna102@0(leader).log v19480 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19480)
>2018-07-11 05:57:35.511153 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 05:57:35.511215 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 490/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:35.674615 7f29149e4700  1 mon.magna102@0(leader).log v19481 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 19481)
>2018-07-11 05:57:36.815644 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:36.815668 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3521186329' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:36.933097 7f29149e4700  1 mon.magna102@0(leader).log v19482 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19482)
>2018-07-11 05:57:38.869253 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:38.869276 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1139794629' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:39.066805 7f29149e4700  1 mon.magna102@0(leader).log v19483 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19483)
>2018-07-11 05:57:40.349836 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:40.349859 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1523390437' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:40.549459 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 489/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:41.217342 7f29149e4700  1 mon.magna102@0(leader).log v19484 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19484)
>2018-07-11 05:57:43.951124 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:43.951148 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4222863133' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:44.667749 7f29149e4700  1 mon.magna102@0(leader).log v19485 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19485)
>2018-07-11 05:57:45.359335 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:45.359359 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/639082018' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:45.834740 7f29149e4700  1 mon.magna102@0(leader).log v19486 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19486)
>2018-07-11 05:57:47.389162 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:47.389198 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1261573162' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:47.509937 7f29149e4700  1 mon.magna102@0(leader).log v19487 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19487)
>2018-07-11 05:57:48.825434 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 05:57:48.825457 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1742181397' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 05:57:48.943673 7f29149e4700  1 mon.magna102@0(leader).log v19488 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19488)
>2018-07-11 05:57:49.943688 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 488/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:50.094019 7f29149e4700  1 mon.magna102@0(leader).log v19489 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19489)
>2018-07-11 05:57:59.523194 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 487/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:57:59.687818 7f29149e4700  1 mon.magna102@0(leader).log v19490 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19490)
>2018-07-11 05:58:00.351432 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 05:58:09.527633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 486/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:09.688653 7f29149e4700  1 mon.magna102@0(leader).log v19491 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19491)
>2018-07-11 05:58:19.532451 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 485/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:19.681449 7f29149e4700  1 mon.magna102@0(leader).log v19492 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19492)
>2018-07-11 05:58:29.536879 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 484/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:29.690649 7f29149e4700  1 mon.magna102@0(leader).log v19493 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19493)
>2018-07-11 05:58:35.674752 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10265. Immutable memtables: 0.
>
>2018-07-11 05:58:35.674797 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 759] Syncing log #10262
>2018-07-11 05:58:35.691330 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.674787) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 05:58:35.691341 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 759] Flushing memtable with next log file: 10265
>
>2018-07-11 05:58:35.691369 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288715691360, "job": 759, "event": "flush_started", "num_memtables": 1, "num_entries": 2222, "num_deletes": 251, "memory_usage": 4999256}
>2018-07-11 05:58:35.691372 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 759] Level-0 flush table #10266: started
>2018-07-11 05:58:35.766928 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288715766913, "cf_name": "default", "job": 759, "event": "table_file_creation", "file_number": 10266, "file_size": 4696972, "table_properties": {"data_size": 4663474, "index_size": 17843, "filter_size": 14729, "raw_key_size": 18717, "raw_average_key_size": 20, "raw_value_size": 4638055, "raw_average_value_size": 5085, "num_data_blocks": 533, "num_entries": 912, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 05:58:35.766941 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 759] Level-0 flush table #10266: 4696972 bytes OK
>2018-07-11 05:58:35.816914 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.783395) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10266 started
>2018-07-11 05:58:35.816922 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.816872) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10266: memtable #1 done
>2018-07-11 05:58:35.816925 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.816887) EVENT_LOG_v1 {"time_micros": 1531288715816880, "job": 759, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 05:58:35.816939 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.816903) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 05:58:35.816946 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 759] Try to delete WAL files size 4951074, prev total WAL file size 4951074, number of live WAL files 2.
>
>2018-07-11 05:58:35.818116 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 05:58:35.818186 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:58:35.818173) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 05:58:35.818193 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 760] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 05:58:35.818200 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 759 Base level 0, inputs: [10266(4586KB)], [10264(13MB)]
>
>2018-07-11 05:58:35.818213 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288715818203, "job": 760, "event": "compaction_started", "files_L0": [10266], "files_L1": [10264], "score": -1, "input_data_size": 19225206}
>2018-07-11 05:58:36.017609 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 760] Generated table #10267: 3513 keys, 17181835 bytes
>2018-07-11 05:58:36.017653 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288716017626, "cf_name": "default", "job": 760, "event": "table_file_creation", "file_number": 10267, "file_size": 17181835, "table_properties": {"data_size": 17052905, "index_size": 73144, "filter_size": 54859, "raw_key_size": 71141, "raw_average_key_size": 20, "raw_value_size": 16951543, "raw_average_value_size": 4825, "num_data_blocks": 2128, "num_entries": 3513, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 05:58:36.050784 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 760] Compacted 1@0 + 1@1 files to L1 => 17181835 bytes
>2018-07-11 05:58:36.084314 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:58:36.084242) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.7 rd, 73.9 wr, level 1, files in(1, 1) out(1) MB in(4.5, 13.9) out(16.4), read-write-amplify(7.8) write-amplify(3.7) OK, records in: 4039, records dropped: 526
>
>2018-07-11 05:58:36.084323 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-05:58:36.084263) EVENT_LOG_v1 {"time_micros": 1531288716084253, "job": 760, "event": "compaction_finished", "compaction_time_micros": 232557, "output_level": 1, "num_output_files": 1, "total_output_size": 17181835, "num_input_records": 4039, "num_output_records": 3513, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 05:58:36.085323 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288716085319, "job": 760, "event": "table_file_deletion", "file_number": 10266}
>2018-07-11 05:58:36.088110 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288716088107, "job": 760, "event": "table_file_deletion", "file_number": 10264}
>2018-07-11 05:58:39.541549 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 483/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:39.693529 7f29149e4700  1 mon.magna102@0(leader).log v19494 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19494)
>2018-07-11 05:58:49.545998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 482/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:49.695681 7f29149e4700  1 mon.magna102@0(leader).log v19495 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19495)
>2018-07-11 05:58:55.548590 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 481/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:58:55.704674 7f29149e4700  1 mon.magna102@0(leader).log v19496 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19496)
>2018-07-11 05:59:00.351613 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 05:59:05.553169 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 480/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:05.705522 7f29149e4700  1 mon.magna102@0(leader).log v19497 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19497)
>2018-07-11 05:59:15.557534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 479/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:15.714624 7f29149e4700  1 mon.magna102@0(leader).log v19498 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19498)
>2018-07-11 05:59:25.561981 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 478/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:25.715682 7f29149e4700  1 mon.magna102@0(leader).log v19499 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19499)
>2018-07-11 05:59:35.566407 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 477/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:35.741545 7f29149e4700  1 mon.magna102@0(leader).log v19500 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19500)
>2018-07-11 05:59:45.571003 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 476/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:45.742419 7f29149e4700  1 mon.magna102@0(leader).log v19501 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19501)
>2018-07-11 05:59:50.608947 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 475/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:50.710758 7f29149e4700  1 mon.magna102@0(leader).log v19502 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19502)
>2018-07-11 05:59:59.577233 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 474/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 05:59:59.728331 7f29149e4700  1 mon.magna102@0(leader).log v19503 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19503)
>2018-07-11 06:00:00.351789 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:00:09.581869 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 473/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:09.729566 7f29149e4700  1 mon.magna102@0(leader).log v19504 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19504)
>2018-07-11 06:00:10.679965 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10268. Immutable memtables: 0.
>
>2018-07-11 06:00:10.680008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 761] Syncing log #10265
>2018-07-11 06:00:10.696520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.679999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:00:10.696531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 761] Flushing memtable with next log file: 10268
>
>2018-07-11 06:00:10.696545 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288810696537, "job": 761, "event": "flush_started", "num_memtables": 1, "num_entries": 672, "num_deletes": 250, "memory_usage": 971408}
>2018-07-11 06:00:10.696548 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 761] Level-0 flush table #10269: started
>2018-07-11 06:00:10.738703 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288810738688, "cf_name": "default", "job": 761, "event": "table_file_creation", "file_number": 10269, "file_size": 645458, "table_properties": {"data_size": 639459, "index_size": 2480, "filter_size": 2600, "raw_key_size": 7472, "raw_average_key_size": 21, "raw_value_size": 632961, "raw_average_value_size": 1834, "num_data_blocks": 77, "num_entries": 345, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:00:10.738713 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 761] Level-0 flush table #10269: 645458 bytes OK
>2018-07-11 06:00:10.788735 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.755280) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10269 started
>2018-07-11 06:00:10.788743 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.788699) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10269: memtable #1 done
>2018-07-11 06:00:10.788746 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.788714) EVENT_LOG_v1 {"time_micros": 1531288810788708, "job": 761, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:00:10.788748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.788725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:00:10.788754 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 761] Try to delete WAL files size 956732, prev total WAL file size 956732, number of live WAL files 2.
>
>2018-07-11 06:00:10.789017 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:00:10.789204 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 06:00:10.789211 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 50837.6 total, 685.2 interval
>Cumulative writes: 71K writes, 341K keys, 71K commit groups, 1.0 writes per commit group, ingest: 0.57 GB, 0.01 MB/s
>Cumulative WAL: 71K writes, 71K syncs, 1.00 writes per sync, written: 0.57 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1153 writes, 5226 keys, 1153 commit groups, 1.0 writes per commit group, ingest: 9.58 MB, 0.01 MB/s
>Interval WAL: 1153 writes, 1153 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   630.33 KB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        32       381    0.083       0      0
>  L1      1/1   16.39 MB   0.0      5.9     0.5      5.4       5.4      0.0       0.0  10.8     72.3     66.2        84       379    0.222   1341K   158K
> Sum      2/2   17.00 MB   0.0      5.9     0.5      5.4       5.9      0.5       0.0  11.8     52.5     52.6       116       760    0.152   1341K   158K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.5     53.1     54.2         1        10    0.147     17K   1977
>Uptime(secs): 50837.6 total, 50837.6 interval
>Flush(GB): cumulative 0.502, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.93 GB write, 0.12 MB/s write, 5.93 GB read, 0.12 MB/s read, 115.6 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   630.33 KB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        32       381    0.083       0      0
>  L1      1/1   16.39 MB   0.0      5.9     0.5      5.4       5.4      0.0       0.0  10.8     72.3     66.2        84       379    0.222   1341K   158K
> Sum      2/2   17.00 MB   0.0      5.9     0.5      5.4       5.9      0.5       0.0  11.8     52.5     52.6       116       760    0.152   1341K   158K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 50837.6 total, 50837.6 interval
>Flush(GB): cumulative 0.502, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 5.93 GB write, 0.12 MB/s write, 5.93 GB read, 0.12 MB/s read, 115.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 06:00:10.789231 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:00:10.789225) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:00:10.789236 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 762] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:00:10.789241 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 761 Base level 0, inputs: [10269(630KB)], [10267(16MB)]
>
>2018-07-11 06:00:10.789253 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288810789244, "job": 762, "event": "compaction_started", "files_L0": [10269], "files_L1": [10267], "score": -1, "input_data_size": 17827293}
>2018-07-11 06:00:10.981098 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 762] Generated table #10270: 3391 keys, 16391931 bytes
>2018-07-11 06:00:10.981138 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288810981112, "cf_name": "default", "job": 762, "event": "table_file_creation", "file_number": 10270, "file_size": 16391931, "table_properties": {"data_size": 16270415, "index_size": 68426, "filter_size": 52163, "raw_key_size": 68306, "raw_average_key_size": 20, "raw_value_size": 16174167, "raw_average_value_size": 4769, "num_data_blocks": 1996, "num_entries": 3391, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:00:11.022637 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 762] Compacted 1@0 + 1@1 files to L1 => 16391931 bytes
>2018-07-11 06:00:11.056168 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:00:11.056098) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.4) out(15.6), read-write-amplify(53.0) write-amplify(25.4) OK, records in: 3858, records dropped: 467
>
>2018-07-11 06:00:11.056177 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:00:11.056120) EVENT_LOG_v1 {"time_micros": 1531288811056110, "job": 762, "event": "compaction_finished", "compaction_time_micros": 233367, "output_level": 1, "num_output_files": 1, "total_output_size": 16391931, "num_input_records": 3858, "num_output_records": 3391, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:00:11.056386 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288811056382, "job": 762, "event": "table_file_deletion", "file_number": 10269}
>2018-07-11 06:00:11.059507 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288811059504, "job": 762, "event": "table_file_deletion", "file_number": 10267}
>2018-07-11 06:00:11.059574 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:00:14.932079 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:00:14.932102 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1296277795' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:00:15.065369 7f29149e4700  1 mon.magna102@0(leader).log v19505 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19505)
>2018-07-11 06:00:16.305804 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:00:16.305828 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/38669294' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:00:17.232437 7f29149e4700  1 mon.magna102@0(leader).log v19506 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19506)
>2018-07-11 06:00:17.805413 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:00:17.805436 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1338046218' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:00:18.307591 7f29149e4700  1 mon.magna102@0(leader).log v19507 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19507)
>2018-07-11 06:00:19.587027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 472/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:19.758199 7f29149e4700  1 mon.magna102@0(leader).log v19508 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19508)
>2018-07-11 06:00:21.643062 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:00:21.643088 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1435974125' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:00:21.644309 7f29191ed700  1 mon.magna102@0(leader).log v19508 check_sub sending message to client.? 10.8.128.102:0/1435974125 with 1 entries (version 19508)
>2018-07-11 06:00:21.775161 7f29149e4700  1 mon.magna102@0(leader).log v19509 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19509)
>2018-07-11 06:00:21.775202 7f29149e4700  1 mon.magna102@0(leader).log v19509 check_sub sending message to client.? 10.8.128.102:0/1435974125 with 0 entries (version 19509)
>2018-07-11 06:00:29.591859 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 471/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:29.743841 7f29149e4700  1 mon.magna102@0(leader).log v19510 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19510)
>2018-07-11 06:00:39.596258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 470/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:39.753502 7f29149e4700  1 mon.magna102@0(leader).log v19511 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19511)
>2018-07-11 06:00:45.598959 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 469/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:45.754068 7f29149e4700  1 mon.magna102@0(leader).log v19512 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19512)
>2018-07-11 06:00:55.603277 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 468/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:00:55.754917 7f29149e4700  1 mon.magna102@0(leader).log v19513 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19513)
>2018-07-11 06:01:00.351959 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:01:05.607924 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 467/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:05.764291 7f29149e4700  1 mon.magna102@0(leader).log v19514 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19514)
>2018-07-11 06:01:15.612486 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 466/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:15.765142 7f29149e4700  1 mon.magna102@0(leader).log v19515 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19515)
>2018-07-11 06:01:20.724426 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10271. Immutable memtables: 0.
>
>2018-07-11 06:01:20.724470 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 763] Syncing log #10268
>2018-07-11 06:01:20.740654 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.724460) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:01:20.740665 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 763] Flushing memtable with next log file: 10271
>
>2018-07-11 06:01:20.740692 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288880740684, "job": 763, "event": "flush_started", "num_memtables": 1, "num_entries": 838, "num_deletes": 500, "memory_usage": 808528}
>2018-07-11 06:01:20.740695 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 763] Level-0 flush table #10272: started
>2018-07-11 06:01:20.784090 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288880784075, "cf_name": "default", "job": 763, "event": "table_file_creation", "file_number": 10272, "file_size": 766125, "table_properties": {"data_size": 758448, "index_size": 3143, "filter_size": 3615, "raw_key_size": 12617, "raw_average_key_size": 20, "raw_value_size": 747956, "raw_average_value_size": 1222, "num_data_blocks": 96, "num_entries": 612, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 06:01:20.784100 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 763] Level-0 flush table #10272: 766125 bytes OK
>2018-07-11 06:01:20.834124 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.800617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10272 started
>2018-07-11 06:01:20.834132 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.834088) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10272: memtable #1 done
>2018-07-11 06:01:20.834135 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.834103) EVENT_LOG_v1 {"time_micros": 1531288880834096, "job": 763, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:01:20.834149 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.834114) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:01:20.834157 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 763] Try to delete WAL files size 788181, prev total WAL file size 788181, number of live WAL files 2.
>
>2018-07-11 06:01:20.834413 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:01:20.834461 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:01:20.834454) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 06:01:20.834471 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 764] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:01:20.834477 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 763 Base level 0, inputs: [10272(748KB)], [10270(15MB)]
>
>2018-07-11 06:01:20.834490 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288880834480, "job": 764, "event": "compaction_started", "files_L0": [10272], "files_L1": [10270], "score": -1, "input_data_size": 17158056}
>2018-07-11 06:01:21.018159 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 764] Generated table #10273: 2995 keys, 14857002 bytes
>2018-07-11 06:01:21.018216 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288881018201, "cf_name": "default", "job": 764, "event": "table_file_creation", "file_number": 10273, "file_size": 14857002, "table_properties": {"data_size": 14746665, "index_size": 62248, "filter_size": 47162, "raw_key_size": 60217, "raw_average_key_size": 20, "raw_value_size": 14660421, "raw_average_value_size": 4894, "num_data_blocks": 1827, "num_entries": 2995, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:01:21.051319 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 764] Compacted 1@0 + 1@1 files to L1 => 14857002 bytes
>2018-07-11 06:01:21.084849 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:01:21.084777) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.6) out(14.2), read-write-amplify(41.8) write-amplify(19.4) OK, records in: 4003, records dropped: 1008
>
>2018-07-11 06:01:21.084857 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:01:21.084798) EVENT_LOG_v1 {"time_micros": 1531288881084789, "job": 764, "event": "compaction_finished", "compaction_time_micros": 216813, "output_level": 1, "num_output_files": 1, "total_output_size": 14857002, "num_input_records": 4003, "num_output_records": 2995, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:01:21.085089 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288881085085, "job": 764, "event": "table_file_deletion", "file_number": 10272}
>2018-07-11 06:01:21.088306 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288881088291, "job": 764, "event": "table_file_deletion", "file_number": 10270}
>2018-07-11 06:01:21.088337 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:01:25.617136 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 465/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:25.769307 7f29149e4700  1 mon.magna102@0(leader).log v19516 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19516)
>2018-07-11 06:01:30.691060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 464/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:30.811405 7f29149e4700  1 mon.magna102@0(leader).log v19517 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19517)
>2018-07-11 06:01:39.623509 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 463/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:39.778982 7f29149e4700  1 mon.magna102@0(leader).log v19518 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19518)
>2018-07-11 06:01:49.628003 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 462/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:49.779929 7f29149e4700  1 mon.magna102@0(leader).log v19519 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19519)
>2018-07-11 06:01:59.632412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 461/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:01:59.789102 7f29149e4700  1 mon.magna102@0(leader).log v19520 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19520)
>2018-07-11 06:02:00.352140 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:02:05.635090 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 460/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:05.789717 7f29149e4700  1 mon.magna102@0(leader).log v19521 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19521)
>2018-07-11 06:02:15.639438 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 459/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:15.808481 7f29149e4700  1 mon.magna102@0(leader).log v19522 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19522)
>2018-07-11 06:02:25.643955 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 458/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:25.801092 7f29149e4700  1 mon.magna102@0(leader).log v19523 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19523)
>2018-07-11 06:02:35.648381 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 457/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:35.793633 7f29149e4700  1 mon.magna102@0(leader).log v19524 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19524)
>2018-07-11 06:02:45.652880 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 456/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:45.802819 7f29149e4700  1 mon.magna102@0(leader).log v19525 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19525)
>2018-07-11 06:02:50.694021 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 455/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:50.786605 7f29149e4700  1 mon.magna102@0(leader).log v19526 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19526)
>2018-07-11 06:02:59.659209 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 454/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:02:59.812606 7f29149e4700  1 mon.magna102@0(leader).log v19527 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19527)
>2018-07-11 06:03:00.352334 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:03:09.663631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 453/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:09.813545 7f29149e4700  1 mon.magna102@0(leader).log v19528 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19528)
>2018-07-11 06:03:19.668218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 452/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:19.823999 7f29149e4700  1 mon.magna102@0(leader).log v19529 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19529)
>2018-07-11 06:03:25.670876 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 451/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:25.824571 7f29149e4700  1 mon.magna102@0(leader).log v19530 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19530)
>2018-07-11 06:03:35.675296 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 450/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:35.825409 7f29149e4700  1 mon.magna102@0(leader).log v19531 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19531)
>2018-07-11 06:03:45.679609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 449/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:45.834640 7f29149e4700  1 mon.magna102@0(leader).log v19532 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19532)
>2018-07-11 06:03:55.684045 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 448/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:03:55.835511 7f29149e4700  1 mon.magna102@0(leader).log v19533 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19533)
>2018-07-11 06:04:00.352507 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:04:05.688542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 447/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:05.861376 7f29149e4700  1 mon.magna102@0(leader).log v19534 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19534)
>2018-07-11 06:04:10.727883 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 446/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:10.853618 7f29149e4700  1 mon.magna102@0(leader).log v19535 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19535)
>2018-07-11 06:04:19.694888 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 445/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:19.847447 7f29149e4700  1 mon.magna102@0(leader).log v19536 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19536)
>2018-07-11 06:04:23.322003 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:04:23.322025 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1228217162' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:04:23.439334 7f29149e4700  1 mon.magna102@0(leader).log v19537 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19537)
>2018-07-11 06:04:29.699423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 444/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:29.848457 7f29149e4700  1 mon.magna102@0(leader).log v19538 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19538)
>2018-07-11 06:04:30.608451 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:04:30.608476 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1369433895' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:04:30.609779 7f29191ed700  1 mon.magna102@0(leader).log v19538 check_sub sending message to client.? 10.8.128.102:0/1369433895 with 1 entries (version 19538)
>2018-07-11 06:04:30.915367 7f29149e4700  1 mon.magna102@0(leader).log v19539 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19539)
>2018-07-11 06:04:30.915400 7f29149e4700  1 mon.magna102@0(leader).log v19539 check_sub sending message to client.? 10.8.128.102:0/1369433895 with 0 entries (version 19539)
>2018-07-11 06:04:39.704049 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 443/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:39.857956 7f29149e4700  1 mon.magna102@0(leader).log v19540 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19540)
>2018-07-11 06:04:39.858002 7f29149e4700  1 mon.magna102@0(leader).log v19540 check_sub sending message to client.? 10.8.128.102:0/1369433895 with 1 entries (version 19540)
>2018-07-11 06:04:49.708664 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 442/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:49.858706 7f29149e4700  1 mon.magna102@0(leader).log v19541 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19541)
>2018-07-11 06:04:55.711368 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 441/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:04:55.859367 7f29149e4700  1 mon.magna102@0(leader).log v19542 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19542)
>2018-07-11 06:05:00.352687 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:05:05.715887 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 440/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:05.868512 7f29149e4700  1 mon.magna102@0(leader).log v19543 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19543)
>2018-07-11 06:05:11.853936 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10274. Immutable memtables: 0.
>
>2018-07-11 06:05:11.853981 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 765] Syncing log #10271
>2018-07-11 06:05:11.870514 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.853971) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:05:11.870526 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 765] Flushing memtable with next log file: 10274
>
>2018-07-11 06:05:11.870539 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289111870532, "job": 765, "event": "flush_started", "num_memtables": 1, "num_entries": 1292, "num_deletes": 251, "memory_usage": 2409232}
>2018-07-11 06:05:11.870542 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 765] Level-0 flush table #10275: started
>2018-07-11 06:05:11.929401 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289111929386, "cf_name": "default", "job": 765, "event": "table_file_creation", "file_number": 10275, "file_size": 2303981, "table_properties": {"data_size": 2285684, "index_size": 9598, "filter_size": 7775, "raw_key_size": 11827, "raw_average_key_size": 20, "raw_value_size": 2270806, "raw_average_value_size": 3942, "num_data_blocks": 291, "num_entries": 576, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:05:11.929411 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 765] Level-0 flush table #10275: 2303981 bytes OK
>2018-07-11 06:05:11.987786 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.945943) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10275 started
>2018-07-11 06:05:11.987794 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.987749) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10275: memtable #1 done
>2018-07-11 06:05:11.987797 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.987763) EVENT_LOG_v1 {"time_micros": 1531289111987757, "job": 765, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:05:11.987799 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.987776) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:05:11.987805 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 765] Try to delete WAL files size 2383185, prev total WAL file size 2383185, number of live WAL files 2.
>
>2018-07-11 06:05:11.988413 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:05:11.988466 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:05:11.988459) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:05:11.988475 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 766] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:05:11.988481 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 765 Base level 0, inputs: [10275(2249KB)], [10273(14MB)]
>
>2018-07-11 06:05:11.988494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289111988484, "job": 766, "event": "compaction_started", "files_L0": [10275], "files_L1": [10273], "score": -1, "input_data_size": 17160983}
>2018-07-11 06:05:12.171821 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 766] Generated table #10276: 3061 keys, 15050775 bytes
>2018-07-11 06:05:12.171870 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289112171842, "cf_name": "default", "job": 766, "event": "table_file_creation", "file_number": 10276, "file_size": 15050775, "table_properties": {"data_size": 14938025, "index_size": 63874, "filter_size": 47949, "raw_key_size": 61797, "raw_average_key_size": 20, "raw_value_size": 14849623, "raw_average_value_size": 4851, "num_data_blocks": 1867, "num_entries": 3061, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:05:12.205009 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 766] Compacted 1@0 + 1@1 files to L1 => 15050775 bytes
>2018-07-11 06:05:12.246851 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:05:12.246784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.3 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.2) out(14.4), read-write-amplify(14.0) write-amplify(6.5) OK, records in: 3571, records dropped: 510
>
>2018-07-11 06:05:12.246860 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:05:12.246805) EVENT_LOG_v1 {"time_micros": 1531289112246796, "job": 766, "event": "compaction_finished", "compaction_time_micros": 216500, "output_level": 1, "num_output_files": 1, "total_output_size": 15050775, "num_input_records": 3571, "num_output_records": 3061, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:05:12.247376 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289112247372, "job": 766, "event": "table_file_deletion", "file_number": 10275}
>2018-07-11 06:05:12.250069 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289112250067, "job": 766, "event": "table_file_deletion", "file_number": 10273}
>2018-07-11 06:05:15.720485 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 439/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:15.897799 7f29149e4700  1 mon.magna102@0(leader).log v19544 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19544)
>2018-07-11 06:05:25.724997 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 438/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:25.873732 7f29149e4700  1 mon.magna102@0(leader).log v19545 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19545)
>2018-07-11 06:05:30.764662 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 437/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:30.857509 7f29149e4700  1 mon.magna102@0(leader).log v19546 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19546)
>2018-07-11 06:05:39.731412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 436/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:39.883415 7f29149e4700  1 mon.magna102@0(leader).log v19547 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19547)
>2018-07-11 06:05:49.735878 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 435/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:49.884291 7f29149e4700  1 mon.magna102@0(leader).log v19548 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19548)
>2018-07-11 06:05:59.740306 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 434/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:05:59.894902 7f29149e4700  1 mon.magna102@0(leader).log v19549 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19549)
>2018-07-11 06:06:00.352881 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:06:09.744869 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 433/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:09.895863 7f29149e4700  1 mon.magna102@0(leader).log v19550 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19550)
>2018-07-11 06:06:15.747441 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 432/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:15.904761 7f29149e4700  1 mon.magna102@0(leader).log v19551 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19551)
>2018-07-11 06:06:25.751884 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 431/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:25.905631 7f29149e4700  1 mon.magna102@0(leader).log v19552 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19552)
>2018-07-11 06:06:35.756424 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 430/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:35.923152 7f29149e4700  1 mon.magna102@0(leader).log v19553 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19553)
>2018-07-11 06:06:45.760920 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 429/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:45.932340 7f29149e4700  1 mon.magna102@0(leader).log v19554 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19554)
>2018-07-11 06:06:55.765261 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 428/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:06:55.941556 7f29149e4700  1 mon.magna102@0(leader).log v19555 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19555)
>2018-07-11 06:07:00.353061 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:07:00.808093 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 427/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:00.918251 7f29149e4700  1 mon.magna102@0(leader).log v19556 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19556)
>2018-07-11 06:07:09.771558 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 426/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:09.919140 7f29149e4700  1 mon.magna102@0(leader).log v19557 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19557)
>2018-07-11 06:07:19.776204 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 425/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:19.928336 7f29149e4700  1 mon.magna102@0(leader).log v19558 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19558)
>2018-07-11 06:07:29.780636 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 424/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:29.929213 7f29149e4700  1 mon.magna102@0(leader).log v19559 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19559)
>2018-07-11 06:07:35.783262 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 423/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:35.929784 7f29149e4700  1 mon.magna102@0(leader).log v19560 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19560)
>2018-07-11 06:07:45.787765 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 422/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:45.939086 7f29149e4700  1 mon.magna102@0(leader).log v19561 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19561)
>2018-07-11 06:07:55.792264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 421/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:07:55.939949 7f29149e4700  1 mon.magna102@0(leader).log v19562 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19562)
>2018-07-11 06:08:00.353236 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:08:05.796577 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 420/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:05.950371 7f29149e4700  1 mon.magna102@0(leader).log v19563 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19563)
>2018-07-11 06:08:15.801131 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 419/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:15.967994 7f29149e4700  1 mon.magna102@0(leader).log v19564 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19564)
>2018-07-11 06:08:20.842893 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 418/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:20.968741 7f29149e4700  1 mon.magna102@0(leader).log v19565 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19565)
>2018-07-11 06:08:29.807661 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 417/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:29.977953 7f29149e4700  1 mon.magna102@0(leader).log v19566 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19566)
>2018-07-11 06:08:30.928350 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10277. Immutable memtables: 0.
>
>2018-07-11 06:08:30.928394 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 767] Syncing log #10274
>2018-07-11 06:08:30.944865 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:08:30.928385) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:08:30.944877 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 767] Flushing memtable with next log file: 10277
>
>2018-07-11 06:08:30.944890 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289310944883, "job": 767, "event": "flush_started", "num_memtables": 1, "num_entries": 1132, "num_deletes": 250, "memory_usage": 2047968}
>2018-07-11 06:08:30.944893 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 767] Level-0 flush table #10278: started
>2018-07-11 06:08:30.995417 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289310995403, "cf_name": "default", "job": 767, "event": "table_file_creation", "file_number": 10278, "file_size": 1322479, "table_properties": {"data_size": 1311961, "index_size": 4971, "filter_size": 4627, "raw_key_size": 9244, "raw_average_key_size": 21, "raw_value_size": 1302435, "raw_average_value_size": 3007, "num_data_blocks": 154, "num_entries": 433, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:08:30.995427 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 767] Level-0 flush table #10278: 1322479 bytes OK
>2018-07-11 06:08:31.053807 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.011968) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10278 started
>2018-07-11 06:08:31.053815 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.053771) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10278: memtable #1 done
>2018-07-11 06:08:31.053817 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.053785) EVENT_LOG_v1 {"time_micros": 1531289311053779, "job": 767, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:08:31.053819 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.053797) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:08:31.053826 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 767] Try to delete WAL files size 2025082, prev total WAL file size 2025082, number of live WAL files 2.
>
>2018-07-11 06:08:31.054335 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:08:31.054386 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.054380) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:08:31.054396 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 768] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:08:31.054402 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 767 Base level 0, inputs: [10278(1291KB)], [10276(14MB)]
>
>2018-07-11 06:08:31.054415 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289311054405, "job": 768, "event": "compaction_started", "files_L0": [10278], "files_L1": [10276], "score": -1, "input_data_size": 16373254}
>2018-07-11 06:08:31.237868 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 768] Generated table #10279: 3079 keys, 15295590 bytes
>2018-07-11 06:08:31.237918 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289311237890, "cf_name": "default", "job": 768, "event": "table_file_creation", "file_number": 10279, "file_size": 15295590, "table_properties": {"data_size": 15182712, "index_size": 63508, "filter_size": 48443, "raw_key_size": 61878, "raw_average_key_size": 20, "raw_value_size": 15094393, "raw_average_value_size": 4902, "num_data_blocks": 1863, "num_entries": 3079, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:08:31.271039 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 768] Compacted 1@0 + 1@1 files to L1 => 15295590 bytes
>2018-07-11 06:08:31.312884 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.312815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.6 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.4) out(14.6), read-write-amplify(23.9) write-amplify(11.6) OK, records in: 3494, records dropped: 415
>
>2018-07-11 06:08:31.312893 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:08:31.312836) EVENT_LOG_v1 {"time_micros": 1531289311312827, "job": 768, "event": "compaction_finished", "compaction_time_micros": 216610, "output_level": 1, "num_output_files": 1, "total_output_size": 15295590, "num_input_records": 3494, "num_output_records": 3079, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:08:31.313210 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289311313194, "job": 768, "event": "table_file_deletion", "file_number": 10278}
>2018-07-11 06:08:31.316085 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289311316082, "job": 768, "event": "table_file_deletion", "file_number": 10276}
>2018-07-11 06:08:31.316212 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:08:39.812416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 416/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:39.980869 7f29149e4700  1 mon.magna102@0(leader).log v19567 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19567)
>2018-07-11 06:08:49.816921 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 415/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:49.991352 7f29149e4700  1 mon.magna102@0(leader).log v19568 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19568)
>2018-07-11 06:08:59.821377 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 414/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:08:59.983934 7f29149e4700  1 mon.magna102@0(leader).log v19569 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19569)
>2018-07-11 06:09:00.353421 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:09:05.823874 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 413/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:05.976331 7f29149e4700  1 mon.magna102@0(leader).log v19570 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19570)
>2018-07-11 06:09:14.244631 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:09:14.244654 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2590321446' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:09:14.960374 7f29149e4700  1 mon.magna102@0(leader).log v19571 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19571)
>2018-07-11 06:09:15.960390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 412/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:16.094035 7f29149e4700  1 mon.magna102@0(leader).log v19572 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19572)
>2018-07-11 06:09:17.152496 7f29149e4700  1 mon.magna102@0(leader).log v19573 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19573)
>2018-07-11 06:09:17.182308 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:09:17.182334 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3392931405' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:09:17.183591 7f29191ed700  1 mon.magna102@0(leader).log v19573 check_sub sending message to client.? 10.8.128.102:0/3392931405 with 1 entries (version 19573)
>2018-07-11 06:09:18.294601 7f29149e4700  1 mon.magna102@0(leader).log v19574 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19574)
>2018-07-11 06:09:18.294634 7f29149e4700  1 mon.magna102@0(leader).log v19574 check_sub sending message to client.? 10.8.128.102:0/3392931405 with 0 entries (version 19574)
>2018-07-11 06:09:25.833246 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 411/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:25.987027 7f29149e4700  1 mon.magna102@0(leader).log v19575 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19575)
>2018-07-11 06:09:25.987071 7f29149e4700  1 mon.magna102@0(leader).log v19575 check_sub sending message to client.? 10.8.128.102:0/3392931405 with 1 entries (version 19575)
>2018-07-11 06:09:35.837616 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 410/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:35.987964 7f29149e4700  1 mon.magna102@0(leader).log v19576 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19576)
>2018-07-11 06:09:35.988008 7f29149e4700  1 mon.magna102@0(leader).log v19576 check_sub sending message to client.? 10.8.128.102:0/3392931405 with 1 entries (version 19576)
>2018-07-11 06:09:40.889036 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 409/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:40.971763 7f29149e4700  1 mon.magna102@0(leader).log v19577 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19577)
>2018-07-11 06:09:40.971808 7f29149e4700  1 mon.magna102@0(leader).log v19577 check_sub sending message to client.? 10.8.128.102:0/3392931405 with 1 entries (version 19577)
>2018-07-11 06:09:49.844738 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 408/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:09:50.008027 7f29149e4700  1 mon.magna102@0(leader).log v19578 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19578)
>2018-07-11 06:09:51.029849 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 06:09:51.029876 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/374927846' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 06:09:51.149836 7f29149e4700  1 mon.magna102@0(leader).log v19579 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19579)
>2018-07-11 06:09:59.849609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 407/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:00.009214 7f29149e4700  1 mon.magna102@0(leader).log v19580 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19580)
>2018-07-11 06:10:00.353602 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:10:01.092668 7f29149e4700  1 mon.magna102@0(leader).log v19581 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19581)
>2018-07-11 06:10:09.854303 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 406/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:10.027228 7f29149e4700  1 mon.magna102@0(leader).log v19582 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19582)
>2018-07-11 06:10:11.261902 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 06:10:11.261928 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/70490292' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 06:10:11.394006 7f29149e4700  1 mon.magna102@0(leader).log v19583 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19583)
>2018-07-11 06:10:19.859232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 405/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:20.028247 7f29149e4700  1 mon.magna102@0(leader).log v19584 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19584)
>2018-07-11 06:10:25.861796 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 404/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:26.012152 7f29149e4700  1 mon.magna102@0(leader).log v19585 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19585)
>2018-07-11 06:10:35.866352 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 403/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:36.014534 7f29149e4700  1 mon.magna102@0(leader).log v19586 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19586)
>2018-07-11 06:10:45.870826 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 402/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:46.023722 7f29149e4700  1 mon.magna102@0(leader).log v19587 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19587)
>2018-07-11 06:10:55.875241 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 401/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:10:56.024449 7f29149e4700  1 mon.magna102@0(leader).log v19588 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19588)
>2018-07-11 06:11:00.353785 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:11:05.879685 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 400/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:06.042107 7f29149e4700  1 mon.magna102@0(leader).log v19589 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19589)
>2018-07-11 06:11:15.884173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 399/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:16.034577 7f29149e4700  1 mon.magna102@0(leader).log v19590 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19590)
>2018-07-11 06:11:25.888518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 398/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:26.035588 7f29149e4700  1 mon.magna102@0(leader).log v19591 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19591)
>2018-07-11 06:11:30.926016 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 397/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:31.027711 7f29149e4700  1 mon.magna102@0(leader).log v19592 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19592)
>2018-07-11 06:11:39.894996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 396/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:40.063295 7f29149e4700  1 mon.magna102@0(leader).log v19593 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19593)
>2018-07-11 06:11:49.899414 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 395/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:11:50.064601 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10280. Immutable memtables: 0.
>
>2018-07-11 06:11:50.064658 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 769] Syncing log #10277
>2018-07-11 06:11:50.080879 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.064637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:11:50.080889 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 769] Flushing memtable with next log file: 10280
>
>2018-07-11 06:11:50.080916 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510080908, "job": 769, "event": "flush_started", "num_memtables": 1, "num_entries": 1172, "num_deletes": 251, "memory_usage": 2143744}
>2018-07-11 06:11:50.080919 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 769] Level-0 flush table #10281: started
>2018-07-11 06:11:50.164991 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510164976, "cf_name": "default", "job": 769, "event": "table_file_creation", "file_number": 10281, "file_size": 2051714, "table_properties": {"data_size": 2035362, "index_size": 8454, "filter_size": 6978, "raw_key_size": 11094, "raw_average_key_size": 20, "raw_value_size": 2021786, "raw_average_value_size": 3737, "num_data_blocks": 257, "num_entries": 541, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:11:50.165014 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 769] Level-0 flush table #10281: 2051714 bytes OK
>2018-07-11 06:11:50.198426 7f29149e4700  1 mon.magna102@0(leader).log v19594 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19594)
>2018-07-11 06:11:50.231687 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.198173) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10281 started
>2018-07-11 06:11:50.231696 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.231650) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10281: memtable #1 done
>2018-07-11 06:11:50.231698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.231665) EVENT_LOG_v1 {"time_micros": 1531289510231659, "job": 769, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:11:50.231700 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.231677) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:11:50.231706 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 769] Try to delete WAL files size 2119678, prev total WAL file size 2141538, number of live WAL files 2.
>
>2018-07-11 06:11:50.232225 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:11:50.232408 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 06:11:50.232414 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 51537.1 total, 699.4 interval
>Cumulative writes: 72K writes, 346K keys, 72K commit groups, 1.0 writes per commit group, ingest: 0.58 GB, 0.01 MB/s
>Cumulative WAL: 72K writes, 72K syncs, 1.00 writes per sync, written: 0.58 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 888 writes, 4442 keys, 888 commit groups, 1.0 writes per commit group, ingest: 7.00 MB, 0.01 MB/s
>Interval WAL: 888 writes, 888 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.96 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        32       385    0.083       0      0
>  L1      1/1   14.59 MB   0.0      6.0     0.5      5.5       5.5     -0.0       0.0  10.8     72.3     66.2        85       383    0.222   1355K   160K
> Sum      2/2   16.54 MB   0.0      6.0     0.5      5.5       6.0      0.5       0.0  11.8     52.6     52.6       117       768    0.152   1355K   160K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.6     54.3     53.9         1         8    0.150     14K   2400
>Uptime(secs): 51537.1 total, 51537.1 interval
>Flush(GB): cumulative 0.508, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.00 GB write, 0.12 MB/s write, 6.00 GB read, 0.12 MB/s read, 116.8 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.96 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.3        32       385    0.083       0      0
>  L1      1/1   14.59 MB   0.0      6.0     0.5      5.5       5.5     -0.0       0.0  10.8     72.3     66.2        85       383    0.222   1355K   160K
> Sum      2/2   16.54 MB   0.0      6.0     0.5      5.5       6.0      0.5       0.0  11.8     52.6     52.6       117       768    0.152   1355K   160K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 51537.1 total, 51537.1 interval
>Flush(GB): cumulative 0.508, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.00 GB write, 0.12 MB/s write, 6.00 GB read, 0.12 MB/s read, 116.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 06:11:50.232435 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.232429) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:11:50.232440 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 770] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:11:50.232446 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 769 Base level 0, inputs: [10281(2003KB)], [10279(14MB)]
>
>2018-07-11 06:11:50.232459 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510232449, "job": 770, "event": "compaction_started", "files_L0": [10281], "files_L1": [10279], "score": -1, "input_data_size": 17347304}
>2018-07-11 06:11:50.407435 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 770] Generated table #10282: 3110 keys, 14843373 bytes
>2018-07-11 06:11:50.407483 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510407468, "cf_name": "default", "job": 770, "event": "table_file_creation", "file_number": 10282, "file_size": 14843373, "table_properties": {"data_size": 14730804, "index_size": 63972, "filter_size": 47670, "raw_key_size": 62725, "raw_average_key_size": 20, "raw_value_size": 14641644, "raw_average_value_size": 4707, "num_data_blocks": 1869, "num_entries": 3110, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:11:50.440553 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 770] Compacted 1@0 + 1@1 files to L1 => 14843373 bytes
>2018-07-11 06:11:50.474051 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.473985) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.4 rd, 71.3 wr, level 1, files in(1, 1) out(1) MB in(2.0, 14.6) out(14.2), read-write-amplify(15.7) write-amplify(7.2) OK, records in: 3620, records dropped: 510
>
>2018-07-11 06:11:50.474060 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:11:50.474006) EVENT_LOG_v1 {"time_micros": 1531289510473997, "job": 770, "event": "compaction_finished", "compaction_time_micros": 208078, "output_level": 1, "num_output_files": 1, "total_output_size": 14843373, "num_input_records": 3620, "num_output_records": 3110, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:11:50.474571 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510474567, "job": 770, "event": "table_file_deletion", "file_number": 10281}
>2018-07-11 06:11:50.477443 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289510477428, "job": 770, "event": "table_file_deletion", "file_number": 10279}
>2018-07-11 06:11:59.904118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 394/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:00.067122 7f29149e4700  1 mon.magna102@0(leader).log v19595 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19595)
>2018-07-11 06:12:00.353962 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:12:09.908582 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 393/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:10.076407 7f29149e4700  1 mon.magna102@0(leader).log v19596 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19596)
>2018-07-11 06:12:15.911268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 392/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:16.085347 7f29149e4700  1 mon.magna102@0(leader).log v19597 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19597)
>2018-07-11 06:12:25.915626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 391/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:26.062441 7f29149e4700  1 mon.magna102@0(leader).log v19598 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19598)
>2018-07-11 06:12:35.920156 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 390/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:36.071629 7f29149e4700  1 mon.magna102@0(leader).log v19599 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19599)
>2018-07-11 06:12:45.924612 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 389/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:46.072480 7f29149e4700  1 mon.magna102@0(leader).log v19600 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19600)
>2018-07-11 06:12:55.929196 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 388/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:12:56.081676 7f29149e4700  1 mon.magna102@0(leader).log v19601 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19601)
>2018-07-11 06:13:00.354133 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:13:05.933592 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 387/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:06.082554 7f29149e4700  1 mon.magna102@0(leader).log v19602 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19602)
>2018-07-11 06:13:10.965737 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 386/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:11.074670 7f29149e4700  1 mon.magna102@0(leader).log v19603 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19603)
>2018-07-11 06:13:19.939993 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 385/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:20.092234 7f29149e4700  1 mon.magna102@0(leader).log v19604 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19604)
>2018-07-11 06:13:29.944521 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 384/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:30.102724 7f29149e4700  1 mon.magna102@0(leader).log v19605 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19605)
>2018-07-11 06:13:39.949012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 383/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:40.120296 7f29149e4700  1 mon.magna102@0(leader).log v19606 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19606)
>2018-07-11 06:13:45.951604 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 382/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:46.120978 7f29149e4700  1 mon.magna102@0(leader).log v19607 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19607)
>2018-07-11 06:13:55.956121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 381/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:13:56.130162 7f29149e4700  1 mon.magna102@0(leader).log v19608 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19608)
>2018-07-11 06:14:00.354324 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:14:05.960574 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 380/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:06.122694 7f29149e4700  1 mon.magna102@0(leader).log v19609 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19609)
>2018-07-11 06:14:15.965267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 379/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:16.140227 7f29149e4700  1 mon.magna102@0(leader).log v19610 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19610)
>2018-07-11 06:14:25.969650 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 378/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:26.117313 7f29149e4700  1 mon.magna102@0(leader).log v19611 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19611)
>2018-07-11 06:14:31.007117 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 377/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:31.109408 7f29149e4700  1 mon.magna102@0(leader).log v19612 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19612)
>2018-07-11 06:14:39.976075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 376/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:40.127000 7f29149e4700  1 mon.magna102@0(leader).log v19613 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19613)
>2018-07-11 06:14:49.980565 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 375/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:14:50.136208 7f29149e4700  1 mon.magna102@0(leader).log v19614 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19614)
>2018-07-11 06:14:59.985060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 374/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:00.137073 7f29149e4700  1 mon.magna102@0(leader).log v19615 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19615)
>2018-07-11 06:15:00.354508 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:15:09.989555 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 373/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:10.146356 7f29149e4700  1 mon.magna102@0(leader).log v19616 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19616)
>2018-07-11 06:15:15.992272 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 372/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:16.146928 7f29149e4700  1 mon.magna102@0(leader).log v19617 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19617)
>2018-07-11 06:15:25.996732 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 371/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:26.157382 7f29149e4700  1 mon.magna102@0(leader).log v19618 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19618)
>2018-07-11 06:15:36.001230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 370/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:36.174987 7f29149e4700  1 mon.magna102@0(leader).log v19619 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19619)
>2018-07-11 06:15:46.005602 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 369/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:46.159292 7f29149e4700  1 mon.magna102@0(leader).log v19620 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19620)
>2018-07-11 06:15:56.010143 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 368/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:15:56.160145 7f29149e4700  1 mon.magna102@0(leader).log v19621 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19621)
>2018-07-11 06:16:00.354679 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8789 MB, avail 861 GB
>2018-07-11 06:16:01.043372 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 367/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:01.143923 7f29149e4700  1 mon.magna102@0(leader).log v19622 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19622)
>2018-07-11 06:16:10.016383 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 366/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:10.169827 7f29149e4700  1 mon.magna102@0(leader).log v19623 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19623)
>2018-07-11 06:16:20.020912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 365/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:20.170627 7f29149e4700  1 mon.magna102@0(leader).log v19624 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19624)
>2018-07-11 06:16:30.025345 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 364/24217500 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:30.172799 7f29149e4700  1 mon.magna102@0(leader).log v19625 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19625)
>2018-07-11 06:16:40.029879 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 363/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:40.181988 7f29149e4700  1 mon.magna102@0(leader).log v19626 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19626)
>2018-07-11 06:16:46.032399 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 362/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:46.182613 7f29149e4700  1 mon.magna102@0(leader).log v19627 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19627)
>2018-07-11 06:16:51.141726 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10283. Immutable memtables: 0.
>
>2018-07-11 06:16:51.141769 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 771] Syncing log #10280
>2018-07-11 06:16:51.158191 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.141760) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:16:51.158202 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 771] Flushing memtable with next log file: 10283
>
>2018-07-11 06:16:51.158232 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811158221, "job": 771, "event": "flush_started", "num_memtables": 1, "num_entries": 1577, "num_deletes": 250, "memory_usage": 3062368}
>2018-07-11 06:16:51.158234 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 771] Level-0 flush table #10284: started
>2018-07-11 06:16:51.217069 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811217055, "cf_name": "default", "job": 771, "event": "table_file_creation", "file_number": 10284, "file_size": 1948066, "table_properties": {"data_size": 1933262, "index_size": 7359, "filter_size": 6525, "raw_key_size": 10935, "raw_average_key_size": 21, "raw_value_size": 1920824, "raw_average_value_size": 3715, "num_data_blocks": 228, "num_entries": 517, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:16:51.217080 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 771] Level-0 flush table #10284: 1948066 bytes OK
>2018-07-11 06:16:51.267075 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.233596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10284 started
>2018-07-11 06:16:51.267083 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.267038) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10284: memtable #1 done
>2018-07-11 06:16:51.267085 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.267052) EVENT_LOG_v1 {"time_micros": 1531289811267046, "job": 771, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:16:51.267100 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.267065) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:16:51.267107 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 771] Try to delete WAL files size 3031047, prev total WAL file size 3031047, number of live WAL files 2.
>
>2018-07-11 06:16:51.267884 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:16:51.267950 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.267940) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:16:51.267956 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 772] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:16:51.267962 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 771 Base level 0, inputs: [10284(1902KB)], [10282(14MB)]
>
>2018-07-11 06:16:51.267975 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811267966, "job": 772, "event": "compaction_started", "files_L0": [10284], "files_L1": [10282], "score": -1, "input_data_size": 16791439}
>2018-07-11 06:16:51.459477 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 772] Generated table #10285: 3263 keys, 16064014 bytes
>2018-07-11 06:16:51.459509 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811459495, "cf_name": "default", "job": 772, "event": "table_file_creation", "file_number": 10285, "file_size": 16064014, "table_properties": {"data_size": 15944038, "index_size": 67848, "filter_size": 51201, "raw_key_size": 65619, "raw_average_key_size": 20, "raw_value_size": 15850057, "raw_average_value_size": 4857, "num_data_blocks": 1990, "num_entries": 3263, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:16:51.500981 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 772] Compacted 1@0 + 1@1 files to L1 => 16064014 bytes
>2018-07-11 06:16:51.534511 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.534444) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.1 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(1.9, 14.2) out(15.3), read-write-amplify(16.9) write-amplify(8.2) OK, records in: 3627, records dropped: 364
>
>2018-07-11 06:16:51.534520 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:16:51.534465) EVENT_LOG_v1 {"time_micros": 1531289811534456, "job": 772, "event": "compaction_finished", "compaction_time_micros": 232991, "output_level": 1, "num_output_files": 1, "total_output_size": 16064014, "num_input_records": 3627, "num_output_records": 3263, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:16:51.534974 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811534970, "job": 772, "event": "table_file_deletion", "file_number": 10284}
>2018-07-11 06:16:51.537831 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289811537828, "job": 772, "event": "table_file_deletion", "file_number": 10282}
>2018-07-11 06:16:51.537944 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:16:56.037146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 361/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:16:56.193960 7f29149e4700  1 mon.magna102@0(leader).log v19628 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19628)
>2018-07-11 06:17:00.354850 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:17:06.041524 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 360/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:06.186483 7f29149e4700  1 mon.magna102@0(leader).log v19629 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19629)
>2018-07-11 06:17:16.046159 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 359/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:16.196904 7f29149e4700  1 mon.magna102@0(leader).log v19630 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19630)
>2018-07-11 06:17:26.050506 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 358/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:26.206216 7f29149e4700  1 mon.magna102@0(leader).log v19631 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19631)
>2018-07-11 06:17:31.109486 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 357/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:31.189972 7f29149e4700  1 mon.magna102@0(leader).log v19632 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19632)
>2018-07-11 06:17:40.056892 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 356/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:40.207546 7f29149e4700  1 mon.magna102@0(leader).log v19633 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19633)
>2018-07-11 06:17:50.061362 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 355/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:17:50.216770 7f29149e4700  1 mon.magna102@0(leader).log v19634 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19634)
>2018-07-11 06:18:00.065893 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 354/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:00.217559 7f29149e4700  1 mon.magna102@0(leader).log v19635 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19635)
>2018-07-11 06:18:00.355036 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:18:10.070350 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 353/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:10.227011 7f29149e4700  1 mon.magna102@0(leader).log v19636 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19636)
>2018-07-11 06:18:16.073027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 352/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:16.245488 7f29149e4700  1 mon.magna102@0(leader).log v19637 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19637)
>2018-07-11 06:18:26.077617 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 351/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:26.246425 7f29149e4700  1 mon.magna102@0(leader).log v19638 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19638)
>2018-07-11 06:18:36.082226 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 350/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:36.255610 7f29149e4700  1 mon.magna102@0(leader).log v19639 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19639)
>2018-07-11 06:18:36.339384 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10286. Immutable memtables: 0.
>
>2018-07-11 06:18:36.339429 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 773] Syncing log #10283
>2018-07-11 06:18:36.355909 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.339419) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:18:36.355920 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 773] Flushing memtable with next log file: 10286
>
>2018-07-11 06:18:36.355934 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916355926, "job": 773, "event": "flush_started", "num_memtables": 1, "num_entries": 723, "num_deletes": 251, "memory_usage": 1096840}
>2018-07-11 06:18:36.355940 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 773] Level-0 flush table #10287: started
>2018-07-11 06:18:36.406441 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916406427, "cf_name": "default", "job": 773, "event": "table_file_creation", "file_number": 10287, "file_size": 1046031, "table_properties": {"data_size": 1036902, "index_size": 4334, "filter_size": 3875, "raw_key_size": 8205, "raw_average_key_size": 20, "raw_value_size": 1028275, "raw_average_value_size": 2557, "num_data_blocks": 132, "num_entries": 402, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:18:36.406452 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 773] Level-0 flush table #10287: 1046031 bytes OK
>2018-07-11 06:18:36.456433 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.423026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10287 started
>2018-07-11 06:18:36.456441 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.456395) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10287: memtable #1 done
>2018-07-11 06:18:36.456444 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.456410) EVENT_LOG_v1 {"time_micros": 1531289916456403, "job": 773, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:18:36.456446 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.456422) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:18:36.456452 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 773] Try to delete WAL files size 1081034, prev total WAL file size 1081034, number of live WAL files 2.
>
>2018-07-11 06:18:36.456852 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:18:36.456906 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.456899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:18:36.456915 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 774] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:18:36.456922 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 773 Base level 0, inputs: [10287(1021KB)], [10285(15MB)]
>
>2018-07-11 06:18:36.456935 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916456925, "job": 774, "event": "compaction_started", "files_L0": [10287], "files_L1": [10285], "score": -1, "input_data_size": 17110045}
>2018-07-11 06:18:36.640569 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 774] Generated table #10288: 3155 keys, 15011298 bytes
>2018-07-11 06:18:36.640619 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916640590, "cf_name": "default", "job": 774, "event": "table_file_creation", "file_number": 10288, "file_size": 15011298, "table_properties": {"data_size": 14898227, "index_size": 64042, "filter_size": 48102, "raw_key_size": 63577, "raw_average_key_size": 20, "raw_value_size": 14808385, "raw_average_value_size": 4693, "num_data_blocks": 1870, "num_entries": 3155, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:18:36.673744 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 774] Compacted 1@0 + 1@1 files to L1 => 15011298 bytes
>2018-07-11 06:18:36.707283 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.707213) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.9 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.3) out(14.3), read-write-amplify(30.7) write-amplify(14.4) OK, records in: 3665, records dropped: 510
>
>2018-07-11 06:18:36.707291 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:18:36.707237) EVENT_LOG_v1 {"time_micros": 1531289916707225, "job": 774, "event": "compaction_finished", "compaction_time_micros": 216793, "output_level": 1, "num_output_files": 1, "total_output_size": 15011298, "num_input_records": 3665, "num_output_records": 3155, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:18:36.707548 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916707544, "job": 774, "event": "table_file_deletion", "file_number": 10287}
>2018-07-11 06:18:36.710552 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531289916710549, "job": 774, "event": "table_file_deletion", "file_number": 10285}
>2018-07-11 06:18:46.086897 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 349/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:46.241929 7f29149e4700  1 mon.magna102@0(leader).log v19640 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19640)
>2018-07-11 06:18:56.091373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 348/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:18:56.259454 7f29149e4700  1 mon.magna102@0(leader).log v19641 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19641)
>2018-07-11 06:19:00.355217 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:19:06.095844 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 347/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:06.244882 7f29149e4700  1 mon.magna102@0(leader).log v19642 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19642)
>2018-07-11 06:19:16.100290 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 346/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:16.245850 7f29149e4700  1 mon.magna102@0(leader).log v19643 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19643)
>2018-07-11 06:19:26.104727 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 345/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:26.255033 7f29149e4700  1 mon.magna102@0(leader).log v19644 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19644)
>2018-07-11 06:19:31.138299 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 344/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:31.238813 7f29149e4700  1 mon.magna102@0(leader).log v19645 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19645)
>2018-07-11 06:19:40.111102 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 343/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:40.264705 7f29149e4700  1 mon.magna102@0(leader).log v19646 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19646)
>2018-07-11 06:19:45.636967 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:45.636990 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1919320650' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:45.756821 7f29149e4700  1 mon.magna102@0(leader).log v19647 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19647)
>2018-07-11 06:19:47.405627 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:47.405651 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/344323812' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:47.890482 7f29149e4700  1 mon.magna102@0(leader).log v19648 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19648)
>2018-07-11 06:19:48.762388 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:48.762411 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2819967342' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:49.032657 7f29149e4700  1 mon.magna102@0(leader).log v19649 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19649)
>2018-07-11 06:19:50.116080 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 342/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:19:50.157821 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:50.157842 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1737155263' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:50.274770 7f29149e4700  1 mon.magna102@0(leader).log v19650 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19650)
>2018-07-11 06:19:51.632742 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:51.632765 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1099463956' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:52.400263 7f29149e4700  1 mon.magna102@0(leader).log v19651 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19651)
>2018-07-11 06:19:53.171690 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:53.171713 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2996658104' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:53.475477 7f29149e4700  1 mon.magna102@0(leader).log v19652 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19652)
>2018-07-11 06:19:54.701066 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:19:54.701089 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2230266371' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:19:55.610377 7f29149e4700  1 mon.magna102@0(leader).log v19653 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19653)
>2018-07-11 06:20:00.121194 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 341/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:00.277701 7f29149e4700  1 mon.magna102@0(leader).log v19654 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19654)
>2018-07-11 06:20:00.355362 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:20:06.124106 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 340/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:06.295523 7f29149e4700  1 mon.magna102@0(leader).log v19655 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19655)
>2018-07-11 06:20:16.128423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 339/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:16.279686 7f29149e4700  1 mon.magna102@0(leader).log v19656 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19656)
>2018-07-11 06:20:26.132963 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 338/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:26.305634 7f29149e4700  1 mon.magna102@0(leader).log v19657 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19657)
>2018-07-11 06:20:36.137378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 337/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:36.306636 7f29149e4700  1 mon.magna102@0(leader).log v19658 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19658)
>2018-07-11 06:20:46.141907 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 336/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:46.299157 7f29149e4700  1 mon.magna102@0(leader).log v19659 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19659)
>2018-07-11 06:20:56.146308 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 335/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:20:56.301293 7f29149e4700  1 mon.magna102@0(leader).log v19660 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19660)
>2018-07-11 06:21:00.355539 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:21:01.267895 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 334/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:01.385125 7f29149e4700  1 mon.magna102@0(leader).log v19661 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19661)
>2018-07-11 06:21:10.152746 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 333/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:10.302689 7f29149e4700  1 mon.magna102@0(leader).log v19662 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19662)
>2018-07-11 06:21:20.157244 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 332/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:20.311987 7f29149e4700  1 mon.magna102@0(leader).log v19663 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19663)
>2018-07-11 06:21:30.161653 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 331/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:30.312850 7f29149e4700  1 mon.magna102@0(leader).log v19664 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19664)
>2018-07-11 06:21:36.164330 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 330/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:36.313544 7f29149e4700  1 mon.magna102@0(leader).log v19665 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19665)
>2018-07-11 06:21:46.168728 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 329/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:46.322746 7f29149e4700  1 mon.magna102@0(leader).log v19666 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19666)
>2018-07-11 06:21:56.173227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 328/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:21:56.341492 7f29149e4700  1 mon.magna102@0(leader).log v19667 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19667)
>2018-07-11 06:22:00.355708 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:22:06.177659 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 327/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:06.350699 7f29149e4700  1 mon.magna102@0(leader).log v19668 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19668)
>2018-07-11 06:22:16.182218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 326/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:16.343236 7f29149e4700  1 mon.magna102@0(leader).log v19669 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19669)
>2018-07-11 06:22:21.309390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 325/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:21.443599 7f29149e4700  1 mon.magna102@0(leader).log v19670 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19670)
>2018-07-11 06:22:30.188462 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 324/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:30.344574 7f29149e4700  1 mon.magna102@0(leader).log v19671 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19671)
>2018-07-11 06:22:40.192971 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 323/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:40.345441 7f29149e4700  1 mon.magna102@0(leader).log v19672 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19672)
>2018-07-11 06:22:50.197381 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 322/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:22:50.346310 7f29149e4700  1 mon.magna102@0(leader).log v19673 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19673)
>2018-07-11 06:23:00.201878 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 321/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:00.355888 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:23:00.356599 7f29149e4700  1 mon.magna102@0(leader).log v19674 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19674)
>2018-07-11 06:23:06.204452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 320/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:06.357327 7f29149e4700  1 mon.magna102@0(leader).log v19675 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19675)
>2018-07-11 06:23:16.208862 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 319/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:16.358203 7f29149e4700  1 mon.magna102@0(leader).log v19676 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19676)
>2018-07-11 06:23:26.213332 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 318/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:26.367549 7f29149e4700  1 mon.magna102@0(leader).log v19677 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19677)
>2018-07-11 06:23:36.217860 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 317/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:36.401770 7f29149e4700  1 mon.magna102@0(leader).log v19678 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19678)
>2018-07-11 06:23:46.222341 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 316/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:46.385958 7f29149e4700  1 mon.magna102@0(leader).log v19679 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19679)
>2018-07-11 06:23:51.352588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 315/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:23:51.486410 7f29149e4700  1 mon.magna102@0(leader).log v19680 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19680)
>2018-07-11 06:24:00.228654 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 314/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:00.356066 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:24:00.371871 7f29149e4700  1 mon.magna102@0(leader).log v19681 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19681)
>2018-07-11 06:24:10.233239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 313/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:10.389521 7f29149e4700  1 mon.magna102@0(leader).log v19682 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19682)
>2018-07-11 06:24:20.237661 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 312/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:20.390374 7f29149e4700  1 mon.magna102@0(leader).log v19683 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19683)
>2018-07-11 06:24:30.242252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 311/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:30.391253 7f29149e4700  1 mon.magna102@0(leader).log v19684 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19684)
>2018-07-11 06:24:40.246655 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 310/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:40.400525 7f29149e4700  1 mon.magna102@0(leader).log v19685 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19685)
>2018-07-11 06:24:46.249329 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 309/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:46.401105 7f29149e4700  1 mon.magna102@0(leader).log v19686 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19686)
>2018-07-11 06:24:56.253852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 308/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:24:56.403235 7f29149e4700  1 mon.magna102@0(leader).log v19687 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19687)
>2018-07-11 06:25:00.356250 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:25:06.258324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 307/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:06.412417 7f29149e4700  1 mon.magna102@0(leader).log v19688 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19688)
>2018-07-11 06:25:11.454866 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10289. Immutable memtables: 0.
>
>2018-07-11 06:25:11.454910 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 775] Syncing log #10286
>2018-07-11 06:25:11.471324 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.454900) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:25:11.471336 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 775] Flushing memtable with next log file: 10289
>
>2018-07-11 06:25:11.471362 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311471354, "job": 775, "event": "flush_started", "num_memtables": 1, "num_entries": 2033, "num_deletes": 250, "memory_usage": 4117192}
>2018-07-11 06:25:11.471366 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 775] Level-0 flush table #10290: started
>2018-07-11 06:25:11.530232 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311530217, "cf_name": "default", "job": 775, "event": "table_file_creation", "file_number": 10290, "file_size": 2642505, "table_properties": {"data_size": 2623016, "index_size": 9953, "filter_size": 8612, "raw_key_size": 12880, "raw_average_key_size": 21, "raw_value_size": 2607369, "raw_average_value_size": 4253, "num_data_blocks": 306, "num_entries": 613, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:25:11.530242 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 775] Level-0 flush table #10290: 2642505 bytes OK
>2018-07-11 06:25:11.580229 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.546738) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10290 started
>2018-07-11 06:25:11.580237 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.580191) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10290: memtable #1 done
>2018-07-11 06:25:11.580239 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.580206) EVENT_LOG_v1 {"time_micros": 1531290311580200, "job": 775, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:25:11.580241 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.580219) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:25:11.580260 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 775] Try to delete WAL files size 4077800, prev total WAL file size 4077800, number of live WAL files 2.
>
>2018-07-11 06:25:11.581261 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:25:11.581442 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 06:25:11.581449 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 52338.4 total, 801.3 interval
>Cumulative writes: 73K writes, 350K keys, 73K commit groups, 1.0 writes per commit group, ingest: 0.58 GB, 0.01 MB/s
>Cumulative WAL: 73K writes, 73K syncs, 1.00 writes per sync, written: 0.58 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 996 writes, 4325 keys, 996 commit groups, 1.0 writes per commit group, ingest: 7.79 MB, 0.01 MB/s
>Interval WAL: 996 writes, 996 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.52 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.4        32       388    0.083       0      0
>  L1      1/1   14.32 MB   0.0      6.0     0.5      5.5       5.5     -0.0       0.0  10.8     72.3     66.2        86       386    0.222   1366K   162K
> Sum      2/2   16.84 MB   0.0      6.0     0.5      5.5       6.0      0.5       0.0  11.8     52.6     52.6       118       774    0.152   1366K   162K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.1     55.8     56.1         1         6    0.146     10K   1384
>Uptime(secs): 52338.4 total, 52338.4 interval
>Flush(GB): cumulative 0.513, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.05 GB write, 0.12 MB/s write, 6.04 GB read, 0.12 MB/s read, 117.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.52 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.4        32       388    0.083       0      0
>  L1      1/1   14.32 MB   0.0      6.0     0.5      5.5       5.5     -0.0       0.0  10.8     72.3     66.2        86       386    0.222   1366K   162K
> Sum      2/2   16.84 MB   0.0      6.0     0.5      5.5       6.0      0.5       0.0  11.8     52.6     52.6       118       774    0.152   1366K   162K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 52338.4 total, 52338.4 interval
>Flush(GB): cumulative 0.513, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.05 GB write, 0.12 MB/s write, 6.04 GB read, 0.12 MB/s read, 117.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 06:25:11.581470 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.581463) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:25:11.581475 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 776] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:25:11.581480 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 775 Base level 0, inputs: [10290(2580KB)], [10288(14MB)]
>
>2018-07-11 06:25:11.581493 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311581484, "job": 776, "event": "compaction_started", "files_L0": [10290], "files_L1": [10288], "score": -1, "input_data_size": 17653803}
>2018-07-11 06:25:11.780963 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 776] Generated table #10291: 3449 keys, 17246928 bytes
>2018-07-11 06:25:11.781008 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311780980, "cf_name": "default", "job": 776, "event": "table_file_creation", "file_number": 10291, "file_size": 17246928, "table_properties": {"data_size": 17119080, "index_size": 72166, "filter_size": 54755, "raw_key_size": 69396, "raw_average_key_size": 20, "raw_value_size": 17019433, "raw_average_value_size": 4934, "num_data_blocks": 2116, "num_entries": 3449, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:25:11.822437 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 776] Compacted 1@0 + 1@1 files to L1 => 17246928 bytes
>2018-07-11 06:25:11.855967 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.855900) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.3 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(2.5, 14.3) out(16.4), read-write-amplify(13.2) write-amplify(6.5) OK, records in: 3768, records dropped: 319
>
>2018-07-11 06:25:11.855976 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:11.855921) EVENT_LOG_v1 {"time_micros": 1531290311855912, "job": 776, "event": "compaction_finished", "compaction_time_micros": 240927, "output_level": 1, "num_output_files": 1, "total_output_size": 17246928, "num_input_records": 3768, "num_output_records": 3449, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:25:11.856593 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311856589, "job": 776, "event": "table_file_deletion", "file_number": 10290}
>2018-07-11 06:25:11.859486 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290311859471, "job": 776, "event": "table_file_deletion", "file_number": 10288}
>2018-07-11 06:25:11.859581 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:25:14.423849 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10292. Immutable memtables: 0.
>
>2018-07-11 06:25:14.423915 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 777] Syncing log #10289
>2018-07-11 06:25:14.440516 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.423905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:25:14.440529 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 777] Flushing memtable with next log file: 10292
>
>2018-07-11 06:25:14.440543 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314440535, "job": 777, "event": "flush_started", "num_memtables": 1, "num_entries": 270, "num_deletes": 251, "memory_usage": 50336}
>2018-07-11 06:25:14.440546 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 777] Level-0 flush table #10293: started
>2018-07-11 06:25:14.474139 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314474126, "cf_name": "default", "job": 777, "event": "table_file_creation", "file_number": 10293, "file_size": 43336, "table_properties": {"data_size": 41467, "index_size": 180, "filter_size": 770, "raw_key_size": 5280, "raw_average_key_size": 20, "raw_value_size": 37848, "raw_average_value_size": 144, "num_data_blocks": 6, "num_entries": 262, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:25:14.474149 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 777] Level-0 flush table #10293: 43336 bytes OK
>2018-07-11 06:25:14.524322 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.490914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10293 started
>2018-07-11 06:25:14.524331 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.524289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10293: memtable #1 done
>2018-07-11 06:25:14.524333 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.524303) EVENT_LOG_v1 {"time_micros": 1531290314524297, "job": 777, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:25:14.524348 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.524314) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:25:14.524353 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 777] Try to delete WAL files size 42563, prev total WAL file size 42563, number of live WAL files 2.
>
>2018-07-11 06:25:14.524455 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:25:14.524503 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.524497) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:25:14.524512 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 778] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:25:14.524518 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 777 Base level 0, inputs: [10293(42KB)], [10291(16MB)]
>
>2018-07-11 06:25:14.524531 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314524521, "job": 778, "event": "compaction_started", "files_L0": [10293], "files_L1": [10291], "score": -1, "input_data_size": 17290264}
>2018-07-11 06:25:14.708361 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 778] Generated table #10294: 3203 keys, 15188148 bytes
>2018-07-11 06:25:14.708399 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314708384, "cf_name": "default", "job": 778, "event": "table_file_creation", "file_number": 10294, "file_size": 15188148, "table_properties": {"data_size": 15074591, "index_size": 64081, "filter_size": 48549, "raw_key_size": 64480, "raw_average_key_size": 20, "raw_value_size": 14984042, "raw_average_value_size": 4678, "num_data_blocks": 1870, "num_entries": 3203, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:25:14.741513 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 778] Compacted 1@0 + 1@1 files to L1 => 15188148 bytes
>2018-07-11 06:25:14.775021 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.774951) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.7 rd, 70.0 wr, level 1, files in(1, 1) out(1) MB in(0.0, 16.4) out(14.5), read-write-amplify(749.5) write-amplify(350.5) OK, records in: 3711, records dropped: 508
>
>2018-07-11 06:25:14.775030 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:25:14.774973) EVENT_LOG_v1 {"time_micros": 1531290314774964, "job": 778, "event": "compaction_finished", "compaction_time_micros": 216968, "output_level": 1, "num_output_files": 1, "total_output_size": 15188148, "num_input_records": 3711, "num_output_records": 3203, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:25:14.775129 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314775124, "job": 778, "event": "table_file_deletion", "file_number": 10293}
>2018-07-11 06:25:14.778263 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290314778260, "job": 778, "event": "table_file_deletion", "file_number": 10291}
>2018-07-11 06:25:16.263301 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 306/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:16.417347 7f29149e4700  1 mon.magna102@0(leader).log v19689 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19689)
>2018-07-11 06:25:26.267595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 305/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:26.418231 7f29149e4700  1 mon.magna102@0(leader).log v19690 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19690)
>2018-07-11 06:25:31.421858 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 304/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:31.536646 7f29149e4700  1 mon.magna102@0(leader).log v19691 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19691)
>2018-07-11 06:25:40.273996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 303/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:40.420877 7f29149e4700  1 mon.magna102@0(leader).log v19692 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19692)
>2018-07-11 06:25:50.278496 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 302/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:25:50.430172 7f29149e4700  1 mon.magna102@0(leader).log v19693 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19693)
>2018-07-11 06:26:00.283003 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 301/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:00.356386 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:26:00.439403 7f29149e4700  1 mon.magna102@0(leader).log v19694 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19694)
>2018-07-11 06:26:06.285518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 300/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:06.448336 7f29149e4700  1 mon.magna102@0(leader).log v19695 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19695)
>2018-07-11 06:26:16.290033 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 299/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:16.440749 7f29149e4700  1 mon.magna102@0(leader).log v19696 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19696)
>2018-07-11 06:26:26.294469 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 298/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:26.450036 7f29149e4700  1 mon.magna102@0(leader).log v19697 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19697)
>2018-07-11 06:26:36.298952 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 297/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:36.468715 7f29149e4700  1 mon.magna102@0(leader).log v19698 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19698)
>2018-07-11 06:26:46.303384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 296/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:46.452989 7f29149e4700  1 mon.magna102@0(leader).log v19699 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19699)
>2018-07-11 06:26:51.427342 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 295/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:26:51.545119 7f29149e4700  1 mon.magna102@0(leader).log v19700 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19700)
>2018-07-11 06:27:00.309715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 294/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:00.356562 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:27:00.454421 7f29149e4700  1 mon.magna102@0(leader).log v19701 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19701)
>2018-07-11 06:27:10.314237 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 293/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:10.472021 7f29149e4700  1 mon.magna102@0(leader).log v19702 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19702)
>2018-07-11 06:27:20.318624 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 292/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:20.472895 7f29149e4700  1 mon.magna102@0(leader).log v19703 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19703)
>2018-07-11 06:27:30.323122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 291/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:30.483384 7f29149e4700  1 mon.magna102@0(leader).log v19704 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19704)
>2018-07-11 06:27:36.325551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 290/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:36.475693 7f29149e4700  1 mon.magna102@0(leader).log v19705 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19705)
>2018-07-11 06:27:46.330111 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 289/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:46.484961 7f29149e4700  1 mon.magna102@0(leader).log v19706 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19706)
>2018-07-11 06:27:56.334511 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 288/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:27:56.485822 7f29149e4700  1 mon.magna102@0(leader).log v19707 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19707)
>2018-07-11 06:28:00.369259 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:28:06.339091 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 287/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:06.511711 7f29149e4700  1 mon.magna102@0(leader).log v19708 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19708)
>2018-07-11 06:28:16.343512 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 286/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:16.504249 7f29149e4700  1 mon.magna102@0(leader).log v19709 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19709)
>2018-07-11 06:28:21.470770 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 285/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:21.604746 7f29149e4700  1 mon.magna102@0(leader).log v19710 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19710)
>2018-07-11 06:28:30.350130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 284/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:30.530650 7f29149e4700  1 mon.magna102@0(leader).log v19711 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19711)
>2018-07-11 06:28:40.354516 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 283/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:40.507741 7f29149e4700  1 mon.magna102@0(leader).log v19712 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19712)
>2018-07-11 06:28:50.359015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 282/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:28:50.500281 7f29149e4700  1 mon.magna102@0(leader).log v19713 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19713)
>2018-07-11 06:29:00.363386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 281/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:00.400783 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:29:00.517819 7f29149e4700  1 mon.magna102@0(leader).log v19714 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19714)
>2018-07-11 06:29:10.367977 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 280/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:10.518770 7f29149e4700  1 mon.magna102@0(leader).log v19715 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19715)
>2018-07-11 06:29:20.372400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 279/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:20.527971 7f29149e4700  1 mon.magna102@0(leader).log v19716 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19716)
>2018-07-11 06:29:26.375093 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 278/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:26.528668 7f29149e4700  1 mon.magna102@0(leader).log v19717 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19717)
>2018-07-11 06:29:36.379490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 277/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:36.564102 7f29149e4700  1 mon.magna102@0(leader).log v19718 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19718)
>2018-07-11 06:29:46.384045 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 276/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:46.556601 7f29149e4700  1 mon.magna102@0(leader).log v19719 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19719)
>2018-07-11 06:29:56.388405 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 275/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:29:56.557492 7f29149e4700  1 mon.magna102@0(leader).log v19720 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19720)
>2018-07-11 06:30:00.424271 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:30:06.393113 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 274/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:06.558670 7f29149e4700  1 mon.magna102@0(leader).log v19721 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19721)
>2018-07-11 06:30:11.516934 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 273/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:11.634213 7f29149e4700  1 mon.magna102@0(leader).log v19722 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19722)
>2018-07-11 06:30:20.399373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 272/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:20.551778 7f29149e4700  1 mon.magna102@0(leader).log v19723 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19723)
>2018-07-11 06:30:30.403877 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 271/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:30.552635 7f29149e4700  1 mon.magna102@0(leader).log v19724 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19724)
>2018-07-11 06:30:40.408458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 270/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:40.563421 7f29149e4700  1 mon.magna102@0(leader).log v19725 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19725)
>2018-07-11 06:30:50.412848 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 269/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:30:50.572572 7f29149e4700  1 mon.magna102@0(leader).log v19726 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19726)
>2018-07-11 06:31:00.417265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 268/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:00.456462 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:31:00.573496 7f29149e4700  1 mon.magna102@0(leader).log v19727 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19727)
>2018-07-11 06:31:10.421701 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 267/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:10.574366 7f29149e4700  1 mon.magna102@0(leader).log v19728 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19728)
>2018-07-11 06:31:20.426212 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 266/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:20.583518 7f29149e4700  1 mon.magna102@0(leader).log v19729 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19729)
>2018-07-11 06:31:30.430638 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 265/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:30.592841 7f29149e4700  1 mon.magna102@0(leader).log v19730 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19730)
>2018-07-11 06:31:36.433159 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 264/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:36.603005 7f29149e4700  1 mon.magna102@0(leader).log v19731 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19731)
>2018-07-11 06:31:46.437582 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 263/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:46.603856 7f29149e4700  1 mon.magna102@0(leader).log v19732 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19732)
>2018-07-11 06:31:56.442127 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 262/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:31:56.588064 7f29149e4700  1 mon.magna102@0(leader).log v19733 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19733)
>2018-07-11 06:32:00.488193 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:32:04.580676 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10295. Immutable memtables: 0.
>
>2018-07-11 06:32:04.580721 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 779] Syncing log #10292
>2018-07-11 06:32:04.597249 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.580711) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:32:04.597261 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 779] Flushing memtable with next log file: 10295
>
>2018-07-11 06:32:04.597288 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724597279, "job": 779, "event": "flush_started", "num_memtables": 1, "num_entries": 2051, "num_deletes": 251, "memory_usage": 4153280}
>2018-07-11 06:32:04.597291 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 779] Level-0 flush table #10296: started
>2018-07-11 06:32:04.672828 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724672801, "cf_name": "default", "job": 779, "event": "table_file_creation", "file_number": 10296, "file_size": 3972036, "table_properties": {"data_size": 3941329, "index_size": 16818, "filter_size": 12963, "raw_key_size": 16602, "raw_average_key_size": 20, "raw_value_size": 3918119, "raw_average_value_size": 4867, "num_data_blocks": 504, "num_entries": 805, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:32:04.672841 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 779] Level-0 flush table #10296: 3972036 bytes OK
>2018-07-11 06:32:04.722802 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.689300) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10296 started
>2018-07-11 06:32:04.722811 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.722762) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10296: memtable #1 done
>2018-07-11 06:32:04.722813 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.722778) EVENT_LOG_v1 {"time_micros": 1531290724722771, "job": 779, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:32:04.722828 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.722792) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:32:04.722835 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 779] Try to delete WAL files size 4113524, prev total WAL file size 4113524, number of live WAL files 2.
>
>2018-07-11 06:32:04.723853 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:32:04.723917 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.723910) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:32:04.723924 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 780] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:32:04.723930 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 779 Base level 0, inputs: [10296(3878KB)], [10294(14MB)]
>
>2018-07-11 06:32:04.723944 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724723934, "job": 780, "event": "compaction_started", "files_L0": [10296], "files_L1": [10294], "score": -1, "input_data_size": 19160184}
>2018-07-11 06:32:04.923529 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 780] Generated table #10297: 3498 keys, 17074761 bytes
>2018-07-11 06:32:04.923576 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724923548, "cf_name": "default", "job": 780, "event": "table_file_creation", "file_number": 10297, "file_size": 17074761, "table_properties": {"data_size": 16946323, "index_size": 72939, "filter_size": 54572, "raw_key_size": 70835, "raw_average_key_size": 20, "raw_value_size": 16845284, "raw_average_value_size": 4815, "num_data_blocks": 2123, "num_entries": 3498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:32:04.956699 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 780] Compacted 1@0 + 1@1 files to L1 => 17074761 bytes
>2018-07-11 06:32:04.990221 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.990147) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.3 rd, 73.4 wr, level 1, files in(1, 1) out(1) MB in(3.8, 14.5) out(16.3), read-write-amplify(9.1) write-amplify(4.3) OK, records in: 4008, records dropped: 510
>
>2018-07-11 06:32:04.990230 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:32:04.990168) EVENT_LOG_v1 {"time_micros": 1531290724990159, "job": 780, "event": "compaction_finished", "compaction_time_micros": 232738, "output_level": 1, "num_output_files": 1, "total_output_size": 17074761, "num_input_records": 4008, "num_output_records": 3498, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:32:04.991038 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724991033, "job": 780, "event": "table_file_deletion", "file_number": 10296}
>2018-07-11 06:32:04.993986 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290724993980, "job": 780, "event": "table_file_deletion", "file_number": 10294}
>2018-07-11 06:32:06.446723 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 261/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:06.599336 7f29149e4700  1 mon.magna102@0(leader).log v19734 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19734)
>2018-07-11 06:32:16.451044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 260/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:16.600193 7f29149e4700  1 mon.magna102@0(leader).log v19735 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19735)
>2018-07-11 06:32:26.455570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 259/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:26.610627 7f29149e4700  1 mon.magna102@0(leader).log v19736 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19736)
>2018-07-11 06:32:36.459924 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 258/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:36.611580 7f29149e4700  1 mon.magna102@0(leader).log v19737 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19737)
>2018-07-11 06:32:46.464348 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 257/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:46.612523 7f29149e4700  1 mon.magna102@0(leader).log v19738 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19738)
>2018-07-11 06:32:51.579069 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 256/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:32:51.696212 7f29149e4700  1 mon.magna102@0(leader).log v19739 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19739)
>2018-07-11 06:33:00.470810 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 255/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:00.505110 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 06:33:00.622097 7f29149e4700  1 mon.magna102@0(leader).log v19740 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19740)
>2018-07-11 06:33:10.475206 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 254/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:10.631456 7f29149e4700  1 mon.magna102@0(leader).log v19741 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19741)
>2018-07-11 06:33:20.479564 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 253/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:20.657409 7f29149e4700  1 mon.magna102@0(leader).log v19742 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19742)
>2018-07-11 06:33:24.182834 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:33:24.182857 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1290424631' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:33:24.317264 7f29149e4700  1 mon.magna102@0(leader).log v19743 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19743)
>2018-07-11 06:33:30.484386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 252/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:30.634656 7f29149e4700  1 mon.magna102@0(leader).log v19744 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19744)
>2018-07-11 06:33:31.651808 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10298. Immutable memtables: 0.
>
>2018-07-11 06:33:31.651853 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 781] Syncing log #10295
>2018-07-11 06:33:31.668303 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.651843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:33:31.668315 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 781] Flushing memtable with next log file: 10298
>
>2018-07-11 06:33:31.668341 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290811668333, "job": 781, "event": "flush_started", "num_memtables": 1, "num_entries": 644, "num_deletes": 250, "memory_usage": 922280}
>2018-07-11 06:33:31.668345 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 781] Level-0 flush table #10299: started
>2018-07-11 06:33:31.710534 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290811710520, "cf_name": "default", "job": 781, "event": "table_file_creation", "file_number": 10299, "file_size": 628420, "table_properties": {"data_size": 622620, "index_size": 2353, "filter_size": 2528, "raw_key_size": 7392, "raw_average_key_size": 21, "raw_value_size": 616271, "raw_average_value_size": 1807, "num_data_blocks": 73, "num_entries": 341, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:33:31.710544 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 781] Level-0 flush table #10299: 628420 bytes OK
>2018-07-11 06:33:31.760545 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.727042) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10299 started
>2018-07-11 06:33:31.760553 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.760509) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10299: memtable #1 done
>2018-07-11 06:33:31.760555 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.760523) EVENT_LOG_v1 {"time_micros": 1531290811760517, "job": 781, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:33:31.760570 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.760535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:33:31.760577 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 781] Try to delete WAL files size 908318, prev total WAL file size 908318, number of live WAL files 2.
>
>2018-07-11 06:33:31.760889 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:33:31.760941 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:33:31.760933) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:33:31.760950 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 782] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:33:31.760957 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 781 Base level 0, inputs: [10299(613KB)], [10297(16MB)]
>
>2018-07-11 06:33:31.760971 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290811760960, "job": 782, "event": "compaction_started", "files_L0": [10299], "files_L1": [10297], "score": -1, "input_data_size": 17703181}
>2018-07-11 06:33:31.952935 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 782] Generated table #10300: 3368 keys, 16248710 bytes
>2018-07-11 06:33:31.952984 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290811952956, "cf_name": "default", "job": 782, "event": "table_file_creation", "file_number": 10300, "file_size": 16248710, "table_properties": {"data_size": 16128133, "index_size": 67919, "filter_size": 51731, "raw_key_size": 67832, "raw_average_key_size": 20, "raw_value_size": 16032534, "raw_average_value_size": 4760, "num_data_blocks": 1982, "num_entries": 3368, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:33:31.986127 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 782] Compacted 1@0 + 1@1 files to L1 => 16248710 bytes
>2018-07-11 06:33:32.019639 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:33:32.019559) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.6 rd, 72.2 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.3) out(15.5), read-write-amplify(54.0) write-amplify(25.9) OK, records in: 3839, records dropped: 471
>
>2018-07-11 06:33:32.019648 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:33:32.019581) EVENT_LOG_v1 {"time_micros": 1531290812019571, "job": 782, "event": "compaction_finished", "compaction_time_micros": 225140, "output_level": 1, "num_output_files": 1, "total_output_size": 16248710, "num_input_records": 3839, "num_output_records": 3368, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:33:32.019860 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290812019856, "job": 782, "event": "table_file_deletion", "file_number": 10299}
>2018-07-11 06:33:32.022959 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290812022956, "job": 782, "event": "table_file_deletion", "file_number": 10297}
>2018-07-11 06:33:32.023076 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:33:32.520039 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:33:32.520063 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2480165649' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:33:32.720280 7f29149e4700  1 mon.magna102@0(leader).log v19745 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19745)
>2018-07-11 06:33:40.489074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 251/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:40.654572 7f29149e4700  1 mon.magna102@0(leader).log v19746 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19746)
>2018-07-11 06:33:50.493431 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 250/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:50.647292 7f29149e4700  1 mon.magna102@0(leader).log v19747 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19747)
>2018-07-11 06:33:56.496262 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 249/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:33:56.664568 7f29149e4700  1 mon.magna102@0(leader).log v19748 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19748)
>2018-07-11 06:34:00.539680 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:34:06.500554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 248/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:06.658367 7f29149e4700  1 mon.magna102@0(leader).log v19749 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19749)
>2018-07-11 06:34:16.505002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 247/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:16.650885 7f29149e4700  1 mon.magna102@0(leader).log v19750 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19750)
>2018-07-11 06:34:26.509295 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 246/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:26.660066 7f29149e4700  1 mon.magna102@0(leader).log v19751 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19751)
>2018-07-11 06:34:36.513707 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 245/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:36.660944 7f29149e4700  1 mon.magna102@0(leader).log v19752 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19752)
>2018-07-11 06:34:46.518176 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 244/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:46.670117 7f29149e4700  1 mon.magna102@0(leader).log v19753 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19753)
>2018-07-11 06:34:51.653852 7f29149e4700  1 mon.magna102@0(leader).log v19754 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19754)
>2018-07-11 06:34:56.522966 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 243/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:34:56.671205 7f29149e4700  1 mon.magna102@0(leader).log v19755 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19755)
>2018-07-11 06:35:00.563131 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:35:01.637823 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 242/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:01.755153 7f29149e4700  1 mon.magna102@0(leader).log v19756 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19756)
>2018-07-11 06:35:10.529292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 241/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:10.690632 7f29149e4700  1 mon.magna102@0(leader).log v19757 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19757)
>2018-07-11 06:35:20.533851 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 240/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:20.799820 7f29149e4700  1 mon.magna102@0(leader).log v19758 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19758)
>2018-07-11 06:35:30.538275 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 239/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:30.717369 7f29149e4700  1 mon.magna102@0(leader).log v19759 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19759)
>2018-07-11 06:35:40.542731 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 238/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:40.718219 7f29149e4700  1 mon.magna102@0(leader).log v19760 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19760)
>2018-07-11 06:35:50.547273 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 237/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:35:50.710714 7f29149e4700  1 mon.magna102@0(leader).log v19761 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19761)
>2018-07-11 06:36:00.551720 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 236/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:00.594615 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:36:00.728306 7f29149e4700  1 mon.magna102@0(leader).log v19762 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19762)
>2018-07-11 06:36:06.640176 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 235/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:07.687221 7f29149e4700  1 mon.magna102@0(leader).log v19763 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19763)
>2018-07-11 06:36:14.558000 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 234/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:14.714315 7f29149e4700  1 mon.magna102@0(leader).log v19764 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19764)
>2018-07-11 06:36:24.562447 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 233/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:24.715261 7f29149e4700  1 mon.magna102@0(leader).log v19765 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19765)
>2018-07-11 06:36:26.716102 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10301. Immutable memtables: 0.
>
>2018-07-11 06:36:26.716146 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 783] Syncing log #10298
>2018-07-11 06:36:26.732342 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.716136) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:36:26.732353 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 783] Flushing memtable with next log file: 10301
>
>2018-07-11 06:36:26.732380 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290986732372, "job": 783, "event": "flush_started", "num_memtables": 1, "num_entries": 1289, "num_deletes": 500, "memory_usage": 1851952}
>2018-07-11 06:36:26.732383 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 783] Level-0 flush table #10302: started
>2018-07-11 06:36:26.782878 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290986782864, "cf_name": "default", "job": 783, "event": "table_file_creation", "file_number": 10302, "file_size": 1764120, "table_properties": {"data_size": 1749161, "index_size": 7327, "filter_size": 6712, "raw_key_size": 15453, "raw_average_key_size": 20, "raw_value_size": 1733727, "raw_average_value_size": 2317, "num_data_blocks": 223, "num_entries": 748, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 06:36:26.782888 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 783] Level-0 flush table #10302: 1764120 bytes OK
>2018-07-11 06:36:26.832903 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.799411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10302 started
>2018-07-11 06:36:26.832912 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.832867) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10302: memtable #1 done
>2018-07-11 06:36:26.832914 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.832881) EVENT_LOG_v1 {"time_micros": 1531290986832875, "job": 783, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:36:26.832916 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.832894) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:36:26.832935 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 783] Try to delete WAL files size 1823368, prev total WAL file size 1823368, number of live WAL files 2.
>
>2018-07-11 06:36:26.833424 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:36:26.833604 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 06:36:26.833611 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 53013.7 total, 675.3 interval
>Cumulative writes: 74K writes, 354K keys, 74K commit groups, 1.0 writes per commit group, ingest: 0.59 GB, 0.01 MB/s
>Cumulative WAL: 74K writes, 74K syncs, 1.00 writes per sync, written: 0.59 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 838 writes, 4254 keys, 838 commit groups, 1.0 writes per commit group, ingest: 6.57 MB, 0.01 MB/s
>Interval WAL: 838 writes, 838 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.68 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.4        32       392    0.083       0      0
>  L1      1/1   15.50 MB   0.0      6.1     0.5      5.6       5.6     -0.0       0.0  10.8     72.3     66.2        87       390    0.222   1382K   163K
> Sum      2/2   17.18 MB   0.0      6.1     0.5      5.6       6.1      0.5       0.0  11.8     52.6     52.6       119       782    0.152   1382K   163K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.3     57.8     58.1         1         8    0.148     15K   1808
>Uptime(secs): 53013.7 total, 53013.7 interval
>Flush(GB): cumulative 0.519, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.11 GB write, 0.12 MB/s write, 6.11 GB read, 0.12 MB/s read, 118.9 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.68 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.4        32       392    0.083       0      0
>  L1      1/1   15.50 MB   0.0      6.1     0.5      5.6       5.6     -0.0       0.0  10.8     72.3     66.2        87       390    0.222   1382K   163K
> Sum      2/2   17.18 MB   0.0      6.1     0.5      5.6       6.1      0.5       0.0  11.8     52.6     52.6       119       782    0.152   1382K   163K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 53013.7 total, 53013.7 interval
>Flush(GB): cumulative 0.519, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.11 GB write, 0.12 MB/s write, 6.11 GB read, 0.12 MB/s read, 118.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 06:36:26.833632 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:36:26.833625) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 06:36:26.833637 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 784] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:36:26.833642 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 783 Base level 0, inputs: [10302(1722KB)], [10300(15MB)]
>
>2018-07-11 06:36:26.833655 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290986833646, "job": 784, "event": "compaction_started", "files_L0": [10302], "files_L1": [10300], "score": -1, "input_data_size": 18012830}
>2018-07-11 06:36:27.016941 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 784] Generated table #10303: 3108 keys, 15724053 bytes
>2018-07-11 06:36:27.016989 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290987016962, "cf_name": "default", "job": 784, "event": "table_file_creation", "file_number": 10303, "file_size": 15724053, "table_properties": {"data_size": 15607207, "index_size": 66066, "filter_size": 49853, "raw_key_size": 62579, "raw_average_key_size": 20, "raw_value_size": 15516660, "raw_average_value_size": 4992, "num_data_blocks": 1940, "num_entries": 3108, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:36:27.050132 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 784] Compacted 1@0 + 1@1 files to L1 => 15724053 bytes
>2018-07-11 06:36:27.083632 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:36:27.083565) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.2 rd, 72.6 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.5) out(15.0), read-write-amplify(19.1) write-amplify(8.9) OK, records in: 4116, records dropped: 1008
>
>2018-07-11 06:36:27.083640 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:36:27.083586) EVENT_LOG_v1 {"time_micros": 1531290987083576, "job": 784, "event": "compaction_finished", "compaction_time_micros": 216461, "output_level": 1, "num_output_files": 1, "total_output_size": 15724053, "num_input_records": 4116, "num_output_records": 3108, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:36:27.084080 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290987084075, "job": 784, "event": "table_file_deletion", "file_number": 10302}
>2018-07-11 06:36:27.087241 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531290987087238, "job": 784, "event": "table_file_deletion", "file_number": 10300}
>2018-07-11 06:36:27.087287 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:36:34.567123 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 232/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:34.709836 7f29149e4700  1 mon.magna102@0(leader).log v19766 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19766)
>2018-07-11 06:36:44.571551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 231/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:44.727374 7f29149e4700  1 mon.magna102@0(leader).log v19767 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19767)
>2018-07-11 06:36:54.575975 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 230/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:36:54.729556 7f29149e4700  1 mon.magna102@0(leader).log v19768 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19768)
>2018-07-11 06:37:00.578583 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 229/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:00.621478 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:37:00.738526 7f29149e4700  1 mon.magna102@0(leader).log v19769 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19769)
>2018-07-11 06:37:10.583121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 228/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:10.747810 7f29149e4700  1 mon.magna102@0(leader).log v19770 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19770)
>2018-07-11 06:37:20.587497 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 227/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:20.757006 7f29149e4700  1 mon.magna102@0(leader).log v19771 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19771)
>2018-07-11 06:37:30.591956 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 226/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:30.766214 7f29149e4700  1 mon.magna102@0(leader).log v19772 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19772)
>2018-07-11 06:37:40.596472 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 225/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:40.742034 7f29149e4700  1 mon.magna102@0(leader).log v19773 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19773)
>2018-07-11 06:37:50.600930 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 224/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:50.751257 7f29149e4700  1 mon.magna102@0(leader).log v19774 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19774)
>2018-07-11 06:37:56.685498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 223/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:37:57.736407 7f29149e4700  1 mon.magna102@0(leader).log v19775 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19775)
>2018-07-11 06:38:00.636602 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:38:04.607239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 222/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:04.762204 7f29149e4700  1 mon.magna102@0(leader).log v19776 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19776)
>2018-07-11 06:38:14.611641 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 221/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:14.788069 7f29149e4700  1 mon.magna102@0(leader).log v19777 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19777)
>2018-07-11 06:38:24.616453 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 220/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:24.797334 7f29149e4700  1 mon.magna102@0(leader).log v19778 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19778)
>2018-07-11 06:38:34.621078 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 219/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:34.773210 7f29149e4700  1 mon.magna102@0(leader).log v19779 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19779)
>2018-07-11 06:38:44.625581 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 218/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:44.782472 7f29149e4700  1 mon.magna102@0(leader).log v19780 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19780)
>2018-07-11 06:38:46.774660 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10304. Immutable memtables: 0.
>
>2018-07-11 06:38:46.774738 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 785] Syncing log #10301
>2018-07-11 06:38:46.791222 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.774728) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:38:46.791233 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 785] Flushing memtable with next log file: 10304
>
>2018-07-11 06:38:46.791247 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291126791239, "job": 785, "event": "flush_started", "num_memtables": 1, "num_entries": 866, "num_deletes": 251, "memory_usage": 1423688}
>2018-07-11 06:38:46.791250 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 785] Level-0 flush table #10305: started
>2018-07-11 06:38:46.841742 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291126841728, "cf_name": "default", "job": 785, "event": "table_file_creation", "file_number": 10305, "file_size": 1358133, "table_properties": {"data_size": 1346682, "index_size": 5680, "filter_size": 4851, "raw_key_size": 9102, "raw_average_key_size": 20, "raw_value_size": 1336466, "raw_average_value_size": 3003, "num_data_blocks": 173, "num_entries": 445, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:38:46.841752 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 785] Level-0 flush table #10305: 1358133 bytes OK
>2018-07-11 06:38:46.900105 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.858317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10305 started
>2018-07-11 06:38:46.900113 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.900069) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10305: memtable #1 done
>2018-07-11 06:38:46.900116 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.900084) EVENT_LOG_v1 {"time_micros": 1531291126900077, "job": 785, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:38:46.900118 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.900095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:38:46.900124 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 785] Try to delete WAL files size 1405031, prev total WAL file size 1405031, number of live WAL files 2.
>
>2018-07-11 06:38:46.900547 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:38:46.900601 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:38:46.900594) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:38:46.900610 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 786] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:38:46.900616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 785 Base level 0, inputs: [10305(1326KB)], [10303(14MB)]
>
>2018-07-11 06:38:46.900629 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291126900619, "job": 786, "event": "compaction_started", "files_L0": [10305], "files_L1": [10303], "score": -1, "input_data_size": 17082186}
>2018-07-11 06:38:47.075828 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 786] Generated table #10306: 3043 keys, 14992479 bytes
>2018-07-11 06:38:47.075877 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291127075850, "cf_name": "default", "job": 786, "event": "table_file_creation", "file_number": 10306, "file_size": 14992479, "table_properties": {"data_size": 14880107, "index_size": 63684, "filter_size": 47761, "raw_key_size": 61434, "raw_average_key_size": 20, "raw_value_size": 14792079, "raw_average_value_size": 4861, "num_data_blocks": 1862, "num_entries": 3043, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:38:47.110245 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 786] Compacted 1@0 + 1@1 files to L1 => 14992479 bytes
>2018-07-11 06:38:47.143775 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:38:47.143705) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.5 rd, 71.5 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.0) out(14.3), read-write-amplify(23.6) write-amplify(11.0) OK, records in: 3553, records dropped: 510
>
>2018-07-11 06:38:47.143784 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:38:47.143726) EVENT_LOG_v1 {"time_micros": 1531291127143717, "job": 786, "event": "compaction_finished", "compaction_time_micros": 209599, "output_level": 1, "num_output_files": 1, "total_output_size": 14992479, "num_input_records": 3553, "num_output_records": 3043, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:38:47.144120 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291127144115, "job": 786, "event": "table_file_deletion", "file_number": 10305}
>2018-07-11 06:38:47.146987 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291127146984, "job": 786, "event": "table_file_deletion", "file_number": 10303}
>2018-07-11 06:38:54.630273 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 217/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:38:54.891271 7f29149e4700  1 mon.magna102@0(leader).log v19781 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19781)
>2018-07-11 06:39:00.670240 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:39:04.634636 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 216/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:04.804216 7f29149e4700  1 mon.magna102@0(leader).log v19782 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19782)
>2018-07-11 06:39:10.637313 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 215/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:10.813125 7f29149e4700  1 mon.magna102@0(leader).log v19783 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19783)
>2018-07-11 06:39:20.642034 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 214/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:20.814057 7f29149e4700  1 mon.magna102@0(leader).log v19784 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19784)
>2018-07-11 06:39:30.646880 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 213/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:30.789982 7f29149e4700  1 mon.magna102@0(leader).log v19785 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19785)
>2018-07-11 06:39:40.651401 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 212/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:40.799115 7f29149e4700  1 mon.magna102@0(leader).log v19786 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19786)
>2018-07-11 06:39:50.655881 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 211/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:39:50.809527 7f29149e4700  1 mon.magna102@0(leader).log v19787 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19787)
>2018-07-11 06:40:00.660303 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:00.693362 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:40:00.810347 7f29149e4700  1 mon.magna102@0(leader).log v19788 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19788)
>2018-07-11 06:40:10.664849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 209/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:10.819704 7f29149e4700  1 mon.magna102@0(leader).log v19789 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19789)
>2018-07-11 06:40:20.669304 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 208/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:20.820628 7f29149e4700  1 mon.magna102@0(leader).log v19790 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19790)
>2018-07-11 06:40:30.673688 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 207/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:30.829800 7f29149e4700  1 mon.magna102@0(leader).log v19791 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19791)
>2018-07-11 06:40:40.678207 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 206/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:40.855987 7f29149e4700  1 mon.magna102@0(leader).log v19792 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19792)
>2018-07-11 06:40:50.682534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 205/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:40:50.858050 7f29149e4700  1 mon.magna102@0(leader).log v19793 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19793)
>2018-07-11 06:41:00.687008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 204/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:00.725246 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:41:00.858911 7f29149e4700  1 mon.magna102@0(leader).log v19794 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19794)
>2018-07-11 06:41:06.725365 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 203/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:06.859508 7f29149e4700  1 mon.magna102@0(leader).log v19795 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19795)
>2018-07-11 06:41:14.693254 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 202/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:14.843496 7f29149e4700  1 mon.magna102@0(leader).log v19796 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19796)
>2018-07-11 06:41:24.697663 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 201/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:24.844462 7f29149e4700  1 mon.magna102@0(leader).log v19797 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19797)
>2018-07-11 06:41:34.702159 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 200/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:34.853623 7f29149e4700  1 mon.magna102@0(leader).log v19798 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19798)
>2018-07-11 06:41:44.706597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 199/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:44.854435 7f29149e4700  1 mon.magna102@0(leader).log v19799 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19799)
>2018-07-11 06:41:51.806437 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10307. Immutable memtables: 0.
>
>2018-07-11 06:41:51.806480 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 787] Syncing log #10304
>2018-07-11 06:41:51.822958 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.806471) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:41:51.822969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 787] Flushing memtable with next log file: 10307
>
>2018-07-11 06:41:51.822982 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291311822975, "job": 787, "event": "flush_started", "num_memtables": 1, "num_entries": 1051, "num_deletes": 250, "memory_usage": 1849024}
>2018-07-11 06:41:51.822985 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 787] Level-0 flush table #10308: started
>2018-07-11 06:41:51.873500 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291311873486, "cf_name": "default", "job": 787, "event": "table_file_creation", "file_number": 10308, "file_size": 1180198, "table_properties": {"data_size": 1170592, "index_size": 4480, "filter_size": 4206, "raw_key_size": 8860, "raw_average_key_size": 21, "raw_value_size": 1161686, "raw_average_value_size": 2806, "num_data_blocks": 139, "num_entries": 414, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:41:51.873511 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 787] Level-0 flush table #10308: 1180198 bytes OK
>2018-07-11 06:41:51.923511 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.890071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10308 started
>2018-07-11 06:41:51.923520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.923475) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10308: memtable #1 done
>2018-07-11 06:41:51.923522 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.923490) EVENT_LOG_v1 {"time_micros": 1531291311923484, "job": 787, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:41:51.923524 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.923502) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:41:51.923530 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 787] Try to delete WAL files size 1827775, prev total WAL file size 1827775, number of live WAL files 2.
>
>2018-07-11 06:41:51.924016 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:41:51.924069 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:41:51.924062) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:41:51.924078 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 788] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:41:51.924084 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 787 Base level 0, inputs: [10308(1152KB)], [10306(14MB)]
>
>2018-07-11 06:41:51.924097 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291311924087, "job": 788, "event": "compaction_started", "files_L0": [10308], "files_L1": [10306], "score": -1, "input_data_size": 16172677}
>2018-07-11 06:41:52.107531 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 788] Generated table #10309: 3035 keys, 15051108 bytes
>2018-07-11 06:41:52.107580 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291312107552, "cf_name": "default", "job": 788, "event": "table_file_creation", "file_number": 10309, "file_size": 15051108, "table_properties": {"data_size": 14939931, "index_size": 62567, "filter_size": 47683, "raw_key_size": 60977, "raw_average_key_size": 20, "raw_value_size": 14852878, "raw_average_value_size": 4893, "num_data_blocks": 1836, "num_entries": 3035, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:41:52.140735 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 788] Compacted 1@0 + 1@1 files to L1 => 15051108 bytes
>2018-07-11 06:41:52.174266 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:41:52.174198) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.7 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.3) out(14.4), read-write-amplify(26.5) write-amplify(12.8) OK, records in: 3457, records dropped: 422
>
>2018-07-11 06:41:52.174275 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:41:52.174220) EVENT_LOG_v1 {"time_micros": 1531291312174210, "job": 788, "event": "compaction_finished", "compaction_time_micros": 216621, "output_level": 1, "num_output_files": 1, "total_output_size": 15051108, "num_input_records": 3457, "num_output_records": 3035, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:41:52.174554 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291312174549, "job": 788, "event": "table_file_deletion", "file_number": 10308}
>2018-07-11 06:41:52.177443 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291312177440, "job": 788, "event": "table_file_deletion", "file_number": 10306}
>2018-07-11 06:41:52.177499 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:41:54.711389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 198/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:41:54.866842 7f29149e4700  1 mon.magna102@0(leader).log v19800 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19800)
>2018-07-11 06:42:00.750579 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:42:04.715863 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 197/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:04.867835 7f29149e4700  1 mon.magna102@0(leader).log v19801 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19801)
>2018-07-11 06:42:14.720337 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 196/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:14.893668 7f29149e4700  1 mon.magna102@0(leader).log v19802 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19802)
>2018-07-11 06:42:20.723016 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 195/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:20.894213 7f29149e4700  1 mon.magna102@0(leader).log v19803 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19803)
>2018-07-11 06:42:30.727500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 194/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:30.903390 7f29149e4700  1 mon.magna102@0(leader).log v19804 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19804)
>2018-07-11 06:42:40.731915 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 193/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:40.888805 7f29149e4700  1 mon.magna102@0(leader).log v19805 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19805)
>2018-07-11 06:42:50.736403 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 192/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:42:50.889701 7f29149e4700  1 mon.magna102@0(leader).log v19806 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19806)
>2018-07-11 06:43:00.740854 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 191/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:00.790259 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:43:00.907349 7f29149e4700  1 mon.magna102@0(leader).log v19807 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19807)
>2018-07-11 06:43:10.745328 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 190/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:10.908106 7f29149e4700  1 mon.magna102@0(leader).log v19808 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19808)
>2018-07-11 06:43:20.749779 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 189/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:20.900761 7f29149e4700  1 mon.magna102@0(leader).log v19809 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19809)
>2018-07-11 06:43:26.804968 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 188/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:26.926366 7f29149e4700  1 mon.magna102@0(leader).log v19810 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19810)
>2018-07-11 06:43:40.758760 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 187/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:40.912001 7f29149e4700  1 mon.magna102@0(leader).log v19811 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19811)
>2018-07-11 06:43:50.763273 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 186/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:50.921173 7f29149e4700  1 mon.magna102@0(leader).log v19812 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19812)
>2018-07-11 06:43:56.806125 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 185/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:43:56.930095 7f29149e4700  1 mon.magna102@0(leader).log v19813 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19813)
>2018-07-11 06:44:00.813399 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:44:00.922042 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:44:00.922066 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3150514629' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:44:01.930554 7f29149e4700  1 mon.magna102@0(leader).log v19814 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19814)
>2018-07-11 06:44:04.770100 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 184/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:04.939403 7f29149e4700  1 mon.magna102@0(leader).log v19815 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19815)
>2018-07-11 06:44:14.774595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 183/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:14.948569 7f29149e4700  1 mon.magna102@0(leader).log v19816 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19816)
>2018-07-11 06:44:16.204352 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:44:16.204375 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/778420649' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:44:16.332042 7f29149e4700  1 mon.magna102@0(leader).log v19817 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19817)
>2018-07-11 06:44:24.779389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 182/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:24.949624 7f29149e4700  1 mon.magna102@0(leader).log v19818 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19818)
>2018-07-11 06:44:34.783995 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 181/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:34.933831 7f29149e4700  1 mon.magna102@0(leader).log v19819 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19819)
>2018-07-11 06:44:40.786540 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 180/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:40.935650 7f29149e4700  1 mon.magna102@0(leader).log v19820 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19820)
>2018-07-11 06:44:50.791184 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 179/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:44:50.944897 7f29149e4700  1 mon.magna102@0(leader).log v19821 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19821)
>2018-07-11 06:45:00.795569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 178/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:00.837015 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:45:00.954099 7f29149e4700  1 mon.magna102@0(leader).log v19822 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19822)
>2018-07-11 06:45:10.800103 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 177/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:10.954915 7f29149e4700  1 mon.magna102@0(leader).log v19823 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19823)
>2018-07-11 06:45:20.804575 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 176/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:20.964067 7f29149e4700  1 mon.magna102@0(leader).log v19824 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19824)
>2018-07-11 06:45:30.809111 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 175/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:30.964916 7f29149e4700  1 mon.magna102@0(leader).log v19825 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19825)
>2018-07-11 06:45:34.948913 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10310. Immutable memtables: 0.
>
>2018-07-11 06:45:34.948957 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 789] Syncing log #10307
>2018-07-11 06:45:34.965466 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:45:34.948947) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:45:34.965477 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 789] Flushing memtable with next log file: 10310
>
>2018-07-11 06:45:34.965504 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291534965496, "job": 789, "event": "flush_started", "num_memtables": 1, "num_entries": 1248, "num_deletes": 251, "memory_usage": 2310872}
>2018-07-11 06:45:34.965507 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 789] Level-0 flush table #10311: started
>2018-07-11 06:45:35.024364 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291535024349, "cf_name": "default", "job": 789, "event": "table_file_creation", "file_number": 10311, "file_size": 2208691, "table_properties": {"data_size": 2191099, "index_size": 9190, "filter_size": 7478, "raw_key_size": 11537, "raw_average_key_size": 20, "raw_value_size": 2176704, "raw_average_value_size": 3873, "num_data_blocks": 279, "num_entries": 562, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:45:35.024374 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 789] Level-0 flush table #10311: 2208691 bytes OK
>2018-07-11 06:45:35.082668 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.040858) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10311 started
>2018-07-11 06:45:35.082676 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.082631) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10311: memtable #1 done
>2018-07-11 06:45:35.082678 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.082645) EVENT_LOG_v1 {"time_micros": 1531291535082639, "job": 789, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:45:35.082693 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.082658) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:45:35.082700 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 789] Try to delete WAL files size 2285087, prev total WAL file size 2285087, number of live WAL files 2.
>
>2018-07-11 06:45:35.083308 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:45:35.083358 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.083351) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:45:35.083367 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 790] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:45:35.083373 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 789 Base level 0, inputs: [10311(2156KB)], [10309(14MB)]
>
>2018-07-11 06:45:35.083387 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291535083377, "job": 790, "event": "compaction_started", "files_L0": [10311], "files_L1": [10309], "score": -1, "input_data_size": 17259799}
>2018-07-11 06:45:35.276263 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 790] Generated table #10312: 3087 keys, 15176195 bytes
>2018-07-11 06:45:35.276307 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291535276291, "cf_name": "default", "job": 790, "event": "table_file_creation", "file_number": 10312, "file_size": 15176195, "table_properties": {"data_size": 15063252, "index_size": 63786, "filter_size": 48230, "raw_key_size": 62267, "raw_average_key_size": 20, "raw_value_size": 14974522, "raw_average_value_size": 4850, "num_data_blocks": 1864, "num_entries": 3087, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:45:35.309457 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 790] Compacted 1@0 + 1@1 files to L1 => 15176195 bytes
>2018-07-11 06:45:35.342968 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.342901) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 67.1 wr, level 1, files in(1, 1) out(1) MB in(2.1, 14.4) out(14.5), read-write-amplify(14.7) write-amplify(6.9) OK, records in: 3597, records dropped: 510
>
>2018-07-11 06:45:35.342977 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:45:35.342922) EVENT_LOG_v1 {"time_micros": 1531291535342912, "job": 790, "event": "compaction_finished", "compaction_time_micros": 226054, "output_level": 1, "num_output_files": 1, "total_output_size": 15176195, "num_input_records": 3597, "num_output_records": 3087, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:45:35.343527 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291535343509, "job": 790, "event": "table_file_deletion", "file_number": 10311}
>2018-07-11 06:45:35.346292 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291535346275, "job": 790, "event": "table_file_deletion", "file_number": 10309}
>2018-07-11 06:45:40.813882 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 174/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:40.960665 7f29149e4700  1 mon.magna102@0(leader).log v19826 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19826)
>2018-07-11 06:45:46.848385 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 173/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:46.969680 7f29149e4700  1 mon.magna102@0(leader).log v19827 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19827)
>2018-07-11 06:45:54.820291 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 172/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:45:54.970383 7f29149e4700  1 mon.magna102@0(leader).log v19828 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19828)
>2018-07-11 06:46:00.854027 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:46:04.824834 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 171/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:04.979670 7f29149e4700  1 mon.magna102@0(leader).log v19829 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19829)
>2018-07-11 06:46:14.829321 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 170/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:14.980484 7f29149e4700  1 mon.magna102@0(leader).log v19830 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19830)
>2018-07-11 06:46:24.833723 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 169/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:24.989657 7f29149e4700  1 mon.magna102@0(leader).log v19831 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19831)
>2018-07-11 06:46:34.838237 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 168/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:35.000113 7f29149e4700  1 mon.magna102@0(leader).log v19832 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19832)
>2018-07-11 06:46:35.772194 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 06:46:35.772218 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/346851935' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 06:46:36.083666 7f29149e4700  1 mon.magna102@0(leader).log v19833 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19833)
>2018-07-11 06:46:40.841115 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 167/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:41.009283 7f29149e4700  1 mon.magna102@0(leader).log v19834 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19834)
>2018-07-11 06:46:50.845492 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 166/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:46:51.018439 7f29149e4700  1 mon.magna102@0(leader).log v19835 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19835)
>2018-07-11 06:47:00.850094 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 165/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:00.885613 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:47:01.019378 7f29149e4700  1 mon.magna102@0(leader).log v19836 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19836)
>2018-07-11 06:47:10.854468 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 164/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:11.003520 7f29149e4700  1 mon.magna102@0(leader).log v19837 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19837)
>2018-07-11 06:47:20.858934 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 163/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:21.012688 7f29149e4700  1 mon.magna102@0(leader).log v19838 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19838)
>2018-07-11 06:47:30.863409 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 162/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:31.013514 7f29149e4700  1 mon.magna102@0(leader).log v19839 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19839)
>2018-07-11 06:47:40.867899 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 161/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:41.023847 7f29149e4700  1 mon.magna102@0(leader).log v19840 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19840)
>2018-07-11 06:47:50.872272 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 160/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:47:51.024751 7f29149e4700  1 mon.magna102@0(leader).log v19841 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19841)
>2018-07-11 06:48:00.876667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 159/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:00.908581 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:48:01.025578 7f29149e4700  1 mon.magna102@0(leader).log v19842 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19842)
>2018-07-11 06:48:06.917508 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 158/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:07.034541 7f29149e4700  1 mon.magna102@0(leader).log v19843 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19843)
>2018-07-11 06:48:14.883022 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 157/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:15.035250 7f29149e4700  1 mon.magna102@0(leader).log v19844 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19844)
>2018-07-11 06:48:24.887531 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 156/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:25.044619 7f29149e4700  1 mon.magna102@0(leader).log v19845 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19845)
>2018-07-11 06:48:34.892038 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 155/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:35.045388 7f29149e4700  1 mon.magna102@0(leader).log v19846 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19846)
>2018-07-11 06:48:44.896536 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 154/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:45.072463 7f29149e4700  1 mon.magna102@0(leader).log v19847 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19847)
>2018-07-11 06:48:54.901002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 153/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:48:55.048496 7f29149e4700  1 mon.magna102@0(leader).log v19848 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19848)
>2018-07-11 06:49:00.932105 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:49:04.905488 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 152/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:05.057694 7f29149e4700  1 mon.magna102@0(leader).log v19849 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19849)
>2018-07-11 06:49:10.908075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 151/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:11.058301 7f29149e4700  1 mon.magna102@0(leader).log v19850 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19850)
>2018-07-11 06:49:20.912533 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 150/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:21.059214 7f29149e4700  1 mon.magna102@0(leader).log v19851 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19851)
>2018-07-11 06:49:30.917049 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 149/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:31.068341 7f29149e4700  1 mon.magna102@0(leader).log v19852 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19852)
>2018-07-11 06:49:40.921468 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 148/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:41.070493 7f29149e4700  1 mon.magna102@0(leader).log v19853 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19853)
>2018-07-11 06:49:50.925990 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 147/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:49:51.079656 7f29149e4700  1 mon.magna102@0(leader).log v19854 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19854)
>2018-07-11 06:50:00.930489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 146/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:00.971777 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:50:01.088853 7f29149e4700  1 mon.magna102@0(leader).log v19855 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19855)
>2018-07-11 06:50:10.935200 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 145/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:11.089779 7f29149e4700  1 mon.magna102@0(leader).log v19856 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19856)
>2018-07-11 06:50:12.048527 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10313. Immutable memtables: 0.
>
>2018-07-11 06:50:12.048569 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 791] Syncing log #10310
>2018-07-11 06:50:12.065031 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.048560) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:50:12.065042 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 791] Flushing memtable with next log file: 10313
>
>2018-07-11 06:50:12.065056 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812065048, "job": 791, "event": "flush_started", "num_memtables": 1, "num_entries": 1469, "num_deletes": 250, "memory_usage": 2818080}
>2018-07-11 06:50:12.065058 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 791] Level-0 flush table #10314: started
>2018-07-11 06:50:12.115559 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812115545, "cf_name": "default", "job": 791, "event": "table_file_creation", "file_number": 10314, "file_size": 1795853, "table_properties": {"data_size": 1782092, "index_size": 6772, "filter_size": 6069, "raw_key_size": 10512, "raw_average_key_size": 21, "raw_value_size": 1770374, "raw_average_value_size": 3569, "num_data_blocks": 210, "num_entries": 496, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:50:12.115569 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 791] Level-0 flush table #10314: 1795853 bytes OK
>2018-07-11 06:50:12.165572 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.132120) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10314 started
>2018-07-11 06:50:12.165580 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.165536) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10314: memtable #1 done
>2018-07-11 06:50:12.165582 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.165550) EVENT_LOG_v1 {"time_micros": 1531291812165544, "job": 791, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:50:12.165584 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.165562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:50:12.165591 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 791] Try to delete WAL files size 2788591, prev total WAL file size 2788591, number of live WAL files 2.
>
>2018-07-11 06:50:12.166255 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:50:12.166434 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 06:50:12.166441 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 53839.0 total, 825.3 interval
>Cumulative writes: 75K writes, 359K keys, 75K commit groups, 1.0 writes per commit group, ingest: 0.60 GB, 0.01 MB/s
>Cumulative WAL: 75K writes, 75K syncs, 1.00 writes per sync, written: 0.60 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1014 writes, 4634 keys, 1014 commit groups, 1.0 writes per commit group, ingest: 7.92 MB, 0.01 MB/s
>Interval WAL: 1014 writes, 1014 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.5        33       396    0.082       0      0
>  L1      1/1   14.47 MB   0.0      6.2     0.5      5.7       5.6     -0.0       0.0  10.8     72.3     66.2        87       394    0.222   1396K   166K
> Sum      2/2   16.19 MB   0.0      6.2     0.5      5.7       6.2      0.5       0.0  11.8     52.7     52.7       120       790    0.152   1396K   166K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.3     57.1     56.2         1         8    0.143     14K   2450
>Uptime(secs): 53839.0 total, 53839.0 interval
>Flush(GB): cumulative 0.526, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.18 GB write, 0.12 MB/s write, 6.17 GB read, 0.12 MB/s read, 120.0 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.5        33       396    0.082       0      0
>  L1      1/1   14.47 MB   0.0      6.2     0.5      5.7       5.6     -0.0       0.0  10.8     72.3     66.2        87       394    0.222   1396K   166K
> Sum      2/2   16.19 MB   0.0      6.2     0.5      5.7       6.2      0.5       0.0  11.8     52.7     52.7       120       790    0.152   1396K   166K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 53839.0 total, 53839.0 interval
>Flush(GB): cumulative 0.526, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.18 GB write, 0.12 MB/s write, 6.17 GB read, 0.12 MB/s read, 120.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 06:50:12.166462 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.166455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:50:12.166467 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 792] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:50:12.166472 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 791 Base level 0, inputs: [10314(1753KB)], [10312(14MB)]
>
>2018-07-11 06:50:12.166484 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812166475, "job": 792, "event": "compaction_started", "files_L0": [10314], "files_L1": [10312], "score": -1, "input_data_size": 16972048}
>2018-07-11 06:50:12.357934 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 792] Generated table #10315: 3207 keys, 16163859 bytes
>2018-07-11 06:50:12.357984 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812357956, "cf_name": "default", "job": 792, "event": "table_file_creation", "file_number": 10315, "file_size": 16163859, "table_properties": {"data_size": 16045295, "index_size": 66612, "filter_size": 51025, "raw_key_size": 64474, "raw_average_key_size": 20, "raw_value_size": 15952949, "raw_average_value_size": 4974, "num_data_blocks": 1954, "num_entries": 3207, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:50:12.391123 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 792] Compacted 1@0 + 1@1 files to L1 => 16163859 bytes
>2018-07-11 06:50:12.424665 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.424598) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.6 rd, 72.0 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.5) out(15.4), read-write-amplify(18.5) write-amplify(9.0) OK, records in: 3583, records dropped: 376
>
>2018-07-11 06:50:12.424674 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:50:12.424619) EVENT_LOG_v1 {"time_micros": 1531291812424610, "job": 792, "event": "compaction_finished", "compaction_time_micros": 224623, "output_level": 1, "num_output_files": 1, "total_output_size": 16163859, "num_input_records": 3583, "num_output_records": 3207, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:50:12.425086 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812425082, "job": 792, "event": "table_file_deletion", "file_number": 10314}
>2018-07-11 06:50:12.427970 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291812427967, "job": 792, "event": "table_file_deletion", "file_number": 10312}
>2018-07-11 06:50:12.428030 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:50:17.015118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 144/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:18.067465 7f29149e4700  1 mon.magna102@0(leader).log v19857 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19857)
>2018-07-11 06:50:24.941871 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 143/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:25.093202 7f29149e4700  1 mon.magna102@0(leader).log v19858 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19858)
>2018-07-11 06:50:34.946324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 142/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:35.103594 7f29149e4700  1 mon.magna102@0(leader).log v19859 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19859)
>2018-07-11 06:50:44.950760 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 141/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:45.121298 7f29149e4700  1 mon.magna102@0(leader).log v19860 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19860)
>2018-07-11 06:50:54.955267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 140/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:50:55.130465 7f29149e4700  1 mon.magna102@0(leader).log v19861 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19861)
>2018-07-11 06:51:00.997340 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:51:04.959708 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 139/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:05.131333 7f29149e4700  1 mon.magna102@0(leader).log v19862 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19862)
>2018-07-11 06:51:10.962405 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 138/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:11.206899 7f29149e4700  1 mon.magna102@0(leader).log v19863 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19863)
>2018-07-11 06:51:20.966852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 137/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:21.141052 7f29149e4700  1 mon.magna102@0(leader).log v19864 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19864)
>2018-07-11 06:51:30.971301 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 136/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:31.118126 7f29149e4700  1 mon.magna102@0(leader).log v19865 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19865)
>2018-07-11 06:51:40.975736 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 135/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:41.127277 7f29149e4700  1 mon.magna102@0(leader).log v19866 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19866)
>2018-07-11 06:51:50.980293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 134/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:51:51.128097 7f29149e4700  1 mon.magna102@0(leader).log v19867 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19867)
>2018-07-11 06:52:00.984657 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 133/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:01.020271 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 06:52:01.129107 7f29149e4700  1 mon.magna102@0(leader).log v19868 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19868)
>2018-07-11 06:52:10.989230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 132/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:11.138248 7f29149e4700  1 mon.magna102@0(leader).log v19869 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19869)
>2018-07-11 06:52:17.030134 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 131/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:17.147190 7f29149e4700  1 mon.magna102@0(leader).log v19870 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19870)
>2018-07-11 06:52:23.139623 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10316. Immutable memtables: 0.
>
>2018-07-11 06:52:23.139668 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 793] Syncing log #10313
>2018-07-11 06:52:23.156187 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.139659) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:52:23.156198 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 793] Flushing memtable with next log file: 10316
>
>2018-07-11 06:52:23.156212 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943156205, "job": 793, "event": "flush_started", "num_memtables": 1, "num_entries": 830, "num_deletes": 251, "memory_usage": 1339616}
>2018-07-11 06:52:23.156215 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 793] Level-0 flush table #10317: started
>2018-07-11 06:52:23.198392 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943198377, "cf_name": "default", "job": 793, "event": "table_file_creation", "file_number": 10317, "file_size": 1277677, "table_properties": {"data_size": 1266805, "index_size": 5347, "filter_size": 4605, "raw_key_size": 8873, "raw_average_key_size": 20, "raw_value_size": 1256986, "raw_average_value_size": 2896, "num_data_blocks": 163, "num_entries": 434, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:52:23.198402 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 793] Level-0 flush table #10317: 1277677 bytes OK
>2018-07-11 06:52:23.256732 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.214907) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10317 started
>2018-07-11 06:52:23.256740 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.256696) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10317: memtable #1 done
>2018-07-11 06:52:23.256743 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.256711) EVENT_LOG_v1 {"time_micros": 1531291943256704, "job": 793, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:52:23.256746 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.256722) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:52:23.256753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 793] Try to delete WAL files size 1321769, prev total WAL file size 1321769, number of live WAL files 2.
>
>2018-07-11 06:52:23.257147 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:52:23.257222 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.257214) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:52:23.257232 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 794] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:52:23.257239 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 793 Base level 0, inputs: [10317(1247KB)], [10315(15MB)]
>
>2018-07-11 06:52:23.257252 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943257242, "job": 794, "event": "compaction_started", "files_L0": [10317], "files_L1": [10315], "score": -1, "input_data_size": 17441536}
>2018-07-11 06:52:23.440771 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 794] Generated table #10318: 3131 keys, 15338530 bytes
>2018-07-11 06:52:23.440814 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943440788, "cf_name": "default", "job": 794, "event": "table_file_creation", "file_number": 10318, "file_size": 15338530, "table_properties": {"data_size": 15225100, "index_size": 63856, "filter_size": 48647, "raw_key_size": 63100, "raw_average_key_size": 20, "raw_value_size": 15135707, "raw_average_value_size": 4834, "num_data_blocks": 1865, "num_entries": 3131, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:52:23.475192 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 794] Compacted 1@0 + 1@1 files to L1 => 15338530 bytes
>2018-07-11 06:52:23.508708 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.508642) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.0 rd, 70.4 wr, level 1, files in(1, 1) out(1) MB in(1.2, 15.4) out(14.6), read-write-amplify(25.7) write-amplify(12.0) OK, records in: 3641, records dropped: 510
>
>2018-07-11 06:52:23.508717 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:52:23.508663) EVENT_LOG_v1 {"time_micros": 1531291943508654, "job": 794, "event": "compaction_finished", "compaction_time_micros": 217917, "output_level": 1, "num_output_files": 1, "total_output_size": 15338530, "num_input_records": 3641, "num_output_records": 3131, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:52:23.509032 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943509028, "job": 794, "event": "table_file_deletion", "file_number": 10317}
>2018-07-11 06:52:23.512009 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531291943512006, "job": 794, "event": "table_file_deletion", "file_number": 10315}
>2018-07-11 06:52:24.995680 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 130/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:25.159471 7f29149e4700  1 mon.magna102@0(leader).log v19871 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19871)
>2018-07-11 06:52:35.000209 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 129/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:35.168645 7f29149e4700  1 mon.magna102@0(leader).log v19872 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19872)
>2018-07-11 06:52:45.004596 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 128/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:45.177983 7f29149e4700  1 mon.magna102@0(leader).log v19873 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19873)
>2018-07-11 06:52:55.009120 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 127/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:52:55.162168 7f29149e4700  1 mon.magna102@0(leader).log v19874 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19874)
>2018-07-11 06:53:01.045824 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:53:05.013580 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 126/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:05.163113 7f29149e4700  1 mon.magna102@0(leader).log v19875 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19875)
>2018-07-11 06:53:15.018109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 125/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:15.163947 7f29149e4700  1 mon.magna102@0(leader).log v19876 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19876)
>2018-07-11 06:53:25.022564 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 124/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:25.174354 7f29149e4700  1 mon.magna102@0(leader).log v19877 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19877)
>2018-07-11 06:53:31.025216 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 123/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:31.174892 7f29149e4700  1 mon.magna102@0(leader).log v19878 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19878)
>2018-07-11 06:53:41.029618 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 122/24217500 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:41.184050 7f29149e4700  1 mon.magna102@0(leader).log v19879 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19879)
>2018-07-11 06:53:51.034252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 121/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:53:51.184899 7f29149e4700  1 mon.magna102@0(leader).log v19880 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19880)
>2018-07-11 06:54:01.038716 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 120/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:01.077099 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:54:01.194096 7f29149e4700  1 mon.magna102@0(leader).log v19881 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19881)
>2018-07-11 06:54:11.043195 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 119/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:11.194926 7f29149e4700  1 mon.magna102@0(leader).log v19882 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19882)
>2018-07-11 06:54:21.047631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 118/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:21.204089 7f29149e4700  1 mon.magna102@0(leader).log v19883 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19883)
>2018-07-11 06:54:31.052122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 117/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:31.222808 7f29149e4700  1 mon.magna102@0(leader).log v19884 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19884)
>2018-07-11 06:54:41.056568 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 116/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:41.223742 7f29149e4700  1 mon.magna102@0(leader).log v19885 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19885)
>2018-07-11 06:54:51.061129 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 115/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:54:51.207992 7f29149e4700  1 mon.magna102@0(leader).log v19886 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19886)
>2018-07-11 06:55:01.065543 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 114/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:01.100146 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:55:01.217193 7f29149e4700  1 mon.magna102@0(leader).log v19887 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19887)
>2018-07-11 06:55:11.070044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 113/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:11.218023 7f29149e4700  1 mon.magna102@0(leader).log v19888 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19888)
>2018-07-11 06:55:17.109956 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 112/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:17.227042 7f29149e4700  1 mon.magna102@0(leader).log v19889 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19889)
>2018-07-11 06:55:25.076422 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 111/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:25.227732 7f29149e4700  1 mon.magna102@0(leader).log v19890 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19890)
>2018-07-11 06:55:35.080909 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 110/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:35.229798 7f29149e4700  1 mon.magna102@0(leader).log v19891 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19891)
>2018-07-11 06:55:45.085387 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 109/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:45.238968 7f29149e4700  1 mon.magna102@0(leader).log v19892 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19892)
>2018-07-11 06:55:55.089783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 108/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:55:55.239848 7f29149e4700  1 mon.magna102@0(leader).log v19893 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19893)
>2018-07-11 06:56:01.131771 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:56:05.094269 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 107/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:05.249057 7f29149e4700  1 mon.magna102@0(leader).log v19894 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19894)
>2018-07-11 06:56:15.098608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 106/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:15.249899 7f29149e4700  1 mon.magna102@0(leader).log v19895 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19895)
>2018-07-11 06:56:25.103195 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 105/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:25.275728 7f29149e4700  1 mon.magna102@0(leader).log v19896 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19896)
>2018-07-11 06:56:31.105763 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 104/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:31.276316 7f29149e4700  1 mon.magna102@0(leader).log v19897 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19897)
>2018-07-11 06:56:41.110250 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 103/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:41.261682 7f29149e4700  1 mon.magna102@0(leader).log v19898 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19898)
>2018-07-11 06:56:51.114749 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 102/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:56:51.262523 7f29149e4700  1 mon.magna102@0(leader).log v19899 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19899)
>2018-07-11 06:57:01.119193 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 101/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:01.154822 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:57:01.271810 7f29149e4700  1 mon.magna102@0(leader).log v19900 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19900)
>2018-07-11 06:57:11.123624 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 100/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:11.272629 7f29149e4700  1 mon.magna102@0(leader).log v19901 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19901)
>2018-07-11 06:57:21.128084 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 99/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:21.281805 7f29149e4700  1 mon.magna102@0(leader).log v19902 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19902)
>2018-07-11 06:57:31.132469 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 98/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:31.282569 7f29149e4700  1 mon.magna102@0(leader).log v19903 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19903)
>2018-07-11 06:57:41.136949 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 97/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:41.293025 7f29149e4700  1 mon.magna102@0(leader).log v19904 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19904)
>2018-07-11 06:57:47.199935 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 96/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:47.318702 7f29149e4700  1 mon.magna102@0(leader).log v19905 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19905)
>2018-07-11 06:57:55.143233 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 95/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:57:55.294401 7f29149e4700  1 mon.magna102@0(leader).log v19906 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19906)
>2018-07-11 06:58:01.186322 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 06:58:05.147633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 94/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:05.311933 7f29149e4700  1 mon.magna102@0(leader).log v19907 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19907)
>2018-07-11 06:58:15.152241 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 93/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:15.321103 7f29149e4700  1 mon.magna102@0(leader).log v19908 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19908)
>2018-07-11 06:58:25.156761 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 92/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:25.330590 7f29149e4700  1 mon.magna102@0(leader).log v19909 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19909)
>2018-07-11 06:58:32.281423 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10319. Immutable memtables: 0.
>
>2018-07-11 06:58:32.281468 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 795] Syncing log #10316
>2018-07-11 06:58:32.297858 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.281457) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:58:32.297869 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 795] Flushing memtable with next log file: 10319
>
>2018-07-11 06:58:32.297896 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312297888, "job": 795, "event": "flush_started", "num_memtables": 1, "num_entries": 1855, "num_deletes": 250, "memory_usage": 3701552}
>2018-07-11 06:58:32.297899 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 795] Level-0 flush table #10320: started
>2018-07-11 06:58:32.356759 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312356745, "cf_name": "default", "job": 795, "event": "table_file_creation", "file_number": 10320, "file_size": 2327519, "table_properties": {"data_size": 2310120, "index_size": 8805, "filter_size": 7670, "raw_key_size": 11920, "raw_average_key_size": 21, "raw_value_size": 2295971, "raw_average_value_size": 4056, "num_data_blocks": 272, "num_entries": 566, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 06:58:32.356770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 795] Level-0 flush table #10320: 2327519 bytes OK
>2018-07-11 06:58:32.416344 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.373262) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10320 started
>2018-07-11 06:58:32.416353 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.416307) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10320: memtable #1 done
>2018-07-11 06:58:32.416355 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.416321) EVENT_LOG_v1 {"time_micros": 1531292312416315, "job": 795, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:58:32.416357 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.416334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:58:32.416376 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 795] Try to delete WAL files size 3665505, prev total WAL file size 3665505, number of live WAL files 2.
>
>2018-07-11 06:58:32.417246 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:58:32.417299 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.417292) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 06:58:32.417309 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 796] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:58:32.417315 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 795 Base level 0, inputs: [10320(2272KB)], [10318(14MB)]
>
>2018-07-11 06:58:32.417328 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312417318, "job": 796, "event": "compaction_started", "files_L0": [10320], "files_L1": [10318], "score": -1, "input_data_size": 17666049}
>2018-07-11 06:58:32.625463 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 796] Generated table #10321: 3367 keys, 17170733 bytes
>2018-07-11 06:58:32.625501 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312625485, "cf_name": "default", "job": 796, "event": "table_file_creation", "file_number": 10321, "file_size": 17170733, "table_properties": {"data_size": 17045311, "index_size": 70412, "filter_size": 54083, "raw_key_size": 67727, "raw_average_key_size": 20, "raw_value_size": 16948031, "raw_average_value_size": 5033, "num_data_blocks": 2065, "num_entries": 3367, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:58:32.666971 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 796] Compacted 1@0 + 1@1 files to L1 => 17170733 bytes
>2018-07-11 06:58:32.700413 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.700341) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 70.8 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.6) out(16.4), read-write-amplify(15.0) write-amplify(7.4) OK, records in: 3697, records dropped: 330
>
>2018-07-11 06:58:32.700422 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:58:32.700363) EVENT_LOG_v1 {"time_micros": 1531292312700353, "job": 796, "event": "compaction_finished", "compaction_time_micros": 249605, "output_level": 1, "num_output_files": 1, "total_output_size": 17170733, "num_input_records": 3697, "num_output_records": 3367, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:58:32.700956 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312700952, "job": 796, "event": "table_file_deletion", "file_number": 10320}
>2018-07-11 06:58:32.703895 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292312703892, "job": 796, "event": "table_file_deletion", "file_number": 10318}
>2018-07-11 06:58:32.704011 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:58:35.161425 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 91/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:35.334628 7f29149e4700  1 mon.magna102@0(leader).log v19910 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19910)
>2018-07-11 06:58:41.164035 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 90/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:41.335236 7f29149e4700  1 mon.magna102@0(leader).log v19911 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19911)
>2018-07-11 06:58:51.168496 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 89/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:58:51.344486 7f29149e4700  1 mon.magna102@0(leader).log v19912 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19912)
>2018-07-11 06:59:01.172951 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 88/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:01.203538 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 06:59:01.320582 7f29149e4700  1 mon.magna102@0(leader).log v19913 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19913)
>2018-07-11 06:59:11.177400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 87/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:11.329733 7f29149e4700  1 mon.magna102@0(leader).log v19914 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19914)
>2018-07-11 06:59:13.313546 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10322. Immutable memtables: 0.
>
>2018-07-11 06:59:13.313589 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 797] Syncing log #10319
>2018-07-11 06:59:13.330134 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.313579) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 06:59:13.330145 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 797] Flushing memtable with next log file: 10322
>
>2018-07-11 06:59:13.330171 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353330163, "job": 797, "event": "flush_started", "num_memtables": 1, "num_entries": 443, "num_deletes": 251, "memory_usage": 452192}
>2018-07-11 06:59:13.330175 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 797] Level-0 flush table #10323: started
>2018-07-11 06:59:13.372327 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353372314, "cf_name": "default", "job": 797, "event": "table_file_creation", "file_number": 10323, "file_size": 428936, "table_properties": {"data_size": 424297, "index_size": 1760, "filter_size": 1960, "raw_key_size": 6434, "raw_average_key_size": 20, "raw_value_size": 418733, "raw_average_value_size": 1320, "num_data_blocks": 54, "num_entries": 317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 06:59:13.372337 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 797] Level-0 flush table #10323: 428936 bytes OK
>2018-07-11 06:59:13.430687 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.388828) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10323 started
>2018-07-11 06:59:13.430695 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.430653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10323: memtable #1 done
>2018-07-11 06:59:13.430698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.430667) EVENT_LOG_v1 {"time_micros": 1531292353430661, "job": 797, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 06:59:13.430700 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.430678) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 06:59:13.430717 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 797] Try to delete WAL files size 441430, prev total WAL file size 441430, number of live WAL files 2.
>
>2018-07-11 06:59:13.430884 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 06:59:13.430933 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.430926) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 06:59:13.430942 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 798] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 06:59:13.430949 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 797 Base level 0, inputs: [10323(418KB)], [10321(16MB)]
>
>2018-07-11 06:59:13.430963 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353430953, "job": 798, "event": "compaction_started", "files_L0": [10323], "files_L1": [10321], "score": -1, "input_data_size": 17599669}
>2018-07-11 06:59:13.614727 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 798] Generated table #10324: 3174 keys, 15513565 bytes
>2018-07-11 06:59:13.614776 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353614749, "cf_name": "default", "job": 798, "event": "table_file_creation", "file_number": 10324, "file_size": 15513565, "table_properties": {"data_size": 15399586, "index_size": 63955, "filter_size": 49097, "raw_key_size": 63914, "raw_average_key_size": 20, "raw_value_size": 15309507, "raw_average_value_size": 4823, "num_data_blocks": 1867, "num_entries": 3174, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 06:59:13.649133 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 798] Compacted 1@0 + 1@1 files to L1 => 15513565 bytes
>2018-07-11 06:59:13.682662 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.682594) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.7 rd, 71.1 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.4) out(14.8), read-write-amplify(77.2) write-amplify(36.2) OK, records in: 3684, records dropped: 510
>
>2018-07-11 06:59:13.682671 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-06:59:13.682616) EVENT_LOG_v1 {"time_micros": 1531292353682607, "job": 798, "event": "compaction_finished", "compaction_time_micros": 218153, "output_level": 1, "num_output_files": 1, "total_output_size": 15513565, "num_input_records": 3684, "num_output_records": 3174, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 06:59:13.682829 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353682824, "job": 798, "event": "table_file_deletion", "file_number": 10323}
>2018-07-11 06:59:13.685959 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292353685956, "job": 798, "event": "table_file_deletion", "file_number": 10321}
>2018-07-11 06:59:21.182101 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 86/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:21.333685 7f29149e4700  1 mon.magna102@0(leader).log v19915 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19915)
>2018-07-11 06:59:31.186588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 85/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:31.342949 7f29149e4700  1 mon.magna102@0(leader).log v19916 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19916)
>2018-07-11 06:59:41.191070 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 84/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:41.360532 7f29149e4700  1 mon.magna102@0(leader).log v19917 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19917)
>2018-07-11 06:59:51.195500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 83/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 06:59:51.369709 7f29149e4700  1 mon.magna102@0(leader).log v19918 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19918)
>2018-07-11 07:00:01.200008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 82/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:01.270414 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:00:01.404149 7f29149e4700  1 mon.magna102@0(leader).log v19919 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19919)
>2018-07-11 07:00:07.242923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 81/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:07.363092 7f29149e4700  1 mon.magna102@0(leader).log v19920 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19920)
>2018-07-11 07:00:15.206309 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 80/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:15.356703 7f29149e4700  1 mon.magna102@0(leader).log v19921 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19921)
>2018-07-11 07:00:25.210677 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 79/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:25.357638 7f29149e4700  1 mon.magna102@0(leader).log v19922 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19922)
>2018-07-11 07:00:35.215151 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 78/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:35.366928 7f29149e4700  1 mon.magna102@0(leader).log v19923 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19923)
>2018-07-11 07:00:45.219572 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 77/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:45.384418 7f29149e4700  1 mon.magna102@0(leader).log v19924 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19924)
>2018-07-11 07:00:55.224138 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 76/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:00:55.376907 7f29149e4700  1 mon.magna102@0(leader).log v19925 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19925)
>2018-07-11 07:01:01.277343 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:01:05.228606 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 75/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:05.378236 7f29149e4700  1 mon.magna102@0(leader).log v19926 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19926)
>2018-07-11 07:01:11.231249 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 74/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:11.388355 7f29149e4700  1 mon.magna102@0(leader).log v19927 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19927)
>2018-07-11 07:01:21.235499 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 73/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:21.405833 7f29149e4700  1 mon.magna102@0(leader).log v19928 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19928)
>2018-07-11 07:01:31.239992 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 72/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:31.415007 7f29149e4700  1 mon.magna102@0(leader).log v19929 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19929)
>2018-07-11 07:01:41.244353 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 71/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:41.407618 7f29149e4700  1 mon.magna102@0(leader).log v19930 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19930)
>2018-07-11 07:01:51.248820 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 70/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:01:51.400099 7f29149e4700  1 mon.magna102@0(leader).log v19931 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19931)
>2018-07-11 07:02:01.253231 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 69/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:01.283960 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:02:01.400963 7f29149e4700  1 mon.magna102@0(leader).log v19932 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19932)
>2018-07-11 07:02:11.257726 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 68/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:11.410320 7f29149e4700  1 mon.magna102@0(leader).log v19933 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19933)
>2018-07-11 07:02:21.262059 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 67/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:21.412396 7f29149e4700  1 mon.magna102@0(leader).log v19934 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19934)
>2018-07-11 07:02:27.304284 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 66/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:27.421320 7f29149e4700  1 mon.magna102@0(leader).log v19935 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19935)
>2018-07-11 07:02:35.268411 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 65/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:35.422013 7f29149e4700  1 mon.magna102@0(leader).log v19936 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19936)
>2018-07-11 07:02:45.272860 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 64/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:45.447836 7f29149e4700  1 mon.magna102@0(leader).log v19937 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19937)
>2018-07-11 07:02:55.277265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 63/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:02:55.440436 7f29149e4700  1 mon.magna102@0(leader).log v19938 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19938)
>2018-07-11 07:03:01.315672 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:03:05.281694 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 62/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:05.457940 7f29149e4700  1 mon.magna102@0(leader).log v19939 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19939)
>2018-07-11 07:03:15.286231 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 61/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:15.458780 7f29149e4700  1 mon.magna102@0(leader).log v19940 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19940)
>2018-07-11 07:03:25.290686 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 60/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:25.435836 7f29149e4700  1 mon.magna102@0(leader).log v19941 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19941)
>2018-07-11 07:03:35.295101 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 59/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:35.444978 7f29149e4700  1 mon.magna102@0(leader).log v19942 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19942)
>2018-07-11 07:03:45.299490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 58/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:45.445841 7f29149e4700  1 mon.magna102@0(leader).log v19943 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19943)
>2018-07-11 07:03:51.302093 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 57/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:03:51.454740 7f29149e4700  1 mon.magna102@0(leader).log v19944 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19944)
>2018-07-11 07:04:01.306395 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 56/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:01.338705 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:04:01.455692 7f29149e4700  1 mon.magna102@0(leader).log v19945 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19945)
>2018-07-11 07:04:11.310924 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 55/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:11.464850 7f29149e4700  1 mon.magna102@0(leader).log v19946 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19946)
>2018-07-11 07:04:21.315236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 54/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:21.465619 7f29149e4700  1 mon.magna102@0(leader).log v19947 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19947)
>2018-07-11 07:04:31.319857 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 53/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:31.476075 7f29149e4700  1 mon.magna102@0(leader).log v19948 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19948)
>2018-07-11 07:04:41.324407 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 52/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:41.485170 7f29149e4700  1 mon.magna102@0(leader).log v19949 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19949)
>2018-07-11 07:04:51.328929 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 51/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:04:51.502726 7f29149e4700  1 mon.magna102@0(leader).log v19950 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19950)
>2018-07-11 07:05:01.333404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 50/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:01.369881 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:05:01.503591 7f29149e4700  1 mon.magna102@0(leader).log v19951 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19951)
>2018-07-11 07:05:11.337855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 49/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:11.496172 7f29149e4700  1 mon.magna102@0(leader).log v19952 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19952)
>2018-07-11 07:05:17.379228 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 48/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:17.496942 7f29149e4700  1 mon.magna102@0(leader).log v19953 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19953)
>2018-07-11 07:05:25.344128 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 47/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:25.498909 7f29149e4700  1 mon.magna102@0(leader).log v19954 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19954)
>2018-07-11 07:05:35.348578 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 46/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:35.499733 7f29149e4700  1 mon.magna102@0(leader).log v19955 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19955)
>2018-07-11 07:05:45.353233 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 45/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:45.500557 7f29149e4700  1 mon.magna102@0(leader).log v19956 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19956)
>2018-07-11 07:05:55.357724 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 44/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:05:55.509733 7f29149e4700  1 mon.magna102@0(leader).log v19957 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19957)
>2018-07-11 07:06:01.393290 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:06:05.362132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 43/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:05.510570 7f29149e4700  1 mon.magna102@0(leader).log v19958 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19958)
>2018-07-11 07:06:05.577786 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10325. Immutable memtables: 0.
>
>2018-07-11 07:06:05.577842 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 799] Syncing log #10322
>2018-07-11 07:06:05.594187 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.577833) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:06:05.594199 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 799] Flushing memtable with next log file: 10325
>
>2018-07-11 07:06:05.594226 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292765594217, "job": 799, "event": "flush_started", "num_memtables": 1, "num_entries": 2050, "num_deletes": 251, "memory_usage": 4146384}
>2018-07-11 07:06:05.594229 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 799] Level-0 flush table #10326: started
>2018-07-11 07:06:05.669609 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292765669595, "cf_name": "default", "job": 799, "event": "table_file_creation", "file_number": 10326, "file_size": 3963808, "table_properties": {"data_size": 3933122, "index_size": 16816, "filter_size": 12944, "raw_key_size": 16583, "raw_average_key_size": 20, "raw_value_size": 3909923, "raw_average_value_size": 4863, "num_data_blocks": 504, "num_entries": 804, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:06:05.669633 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 799] Level-0 flush table #10326: 3963808 bytes OK
>2018-07-11 07:06:05.728067 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.686216) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10326 started
>2018-07-11 07:06:05.728075 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.728026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10326: memtable #1 done
>2018-07-11 07:06:05.728077 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.728041) EVENT_LOG_v1 {"time_micros": 1531292765728034, "job": 799, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:06:05.728092 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.728056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:06:05.728100 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 799] Try to delete WAL files size 4105801, prev total WAL file size 4105801, number of live WAL files 2.
>
>2018-07-11 07:06:05.729079 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:06:05.729279 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 07:06:05.729285 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 54792.6 total, 953.6 interval
>Cumulative writes: 76K writes, 364K keys, 76K commit groups, 1.0 writes per commit group, ingest: 0.61 GB, 0.01 MB/s
>Cumulative WAL: 76K writes, 76K syncs, 1.00 writes per sync, written: 0.61 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1166 writes, 5178 keys, 1166 commit groups, 1.0 writes per commit group, ingest: 9.09 MB, 0.01 MB/s
>Interval WAL: 1166 writes, 1166 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.78 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.6        33       400    0.082       0      0
>  L1      1/1   14.79 MB   0.0      6.2     0.5      5.7       5.7     -0.0       0.0  10.8     72.3     66.2        88       398    0.222   1411K   168K
> Sum      2/2   18.58 MB   0.0      6.2     0.5      5.7       6.2      0.5       0.0  11.7     52.7     52.7       121       798    0.152   1411K   168K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.0     55.6     57.6         1         8    0.149     14K   1726
>Uptime(secs): 54792.6 total, 54792.6 interval
>Flush(GB): cumulative 0.533, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.24 GB write, 0.12 MB/s write, 6.24 GB read, 0.12 MB/s read, 121.2 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.78 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.6        33       400    0.082       0      0
>  L1      1/1   14.79 MB   0.0      6.2     0.5      5.7       5.7     -0.0       0.0  10.8     72.3     66.2        88       398    0.222   1411K   168K
> Sum      2/2   18.58 MB   0.0      6.2     0.5      5.7       6.2      0.5       0.0  11.7     52.7     52.7       121       798    0.152   1411K   168K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 54792.6 total, 54792.6 interval
>Flush(GB): cumulative 0.533, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.24 GB write, 0.12 MB/s write, 6.24 GB read, 0.12 MB/s read, 121.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 07:06:05.729307 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:05.729300) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:06:05.729312 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 800] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:06:05.729317 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 799 Base level 0, inputs: [10326(3870KB)], [10324(14MB)]
>
>2018-07-11 07:06:05.729330 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292765729321, "job": 800, "event": "compaction_started", "files_L0": [10326], "files_L1": [10324], "score": -1, "input_data_size": 19477373}
>2018-07-11 07:06:05.928739 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 800] Generated table #10327: 3468 keys, 17392766 bytes
>2018-07-11 07:06:05.928787 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292765928760, "cf_name": "default", "job": 800, "event": "table_file_creation", "file_number": 10327, "file_size": 17392766, "table_properties": {"data_size": 17263957, "index_size": 72782, "filter_size": 55100, "raw_key_size": 70250, "raw_average_key_size": 20, "raw_value_size": 17163424, "raw_average_value_size": 4949, "num_data_blocks": 2119, "num_entries": 3468, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:06:05.970253 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 800] Compacted 1@0 + 1@1 files to L1 => 17392766 bytes
>2018-07-11 07:06:06.012101 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:06.012035) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.9 rd, 72.2 wr, level 1, files in(1, 1) out(1) MB in(3.8, 14.8) out(16.6), read-write-amplify(9.3) write-amplify(4.4) OK, records in: 3978, records dropped: 510
>
>2018-07-11 07:06:06.012110 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:06.012057) EVENT_LOG_v1 {"time_micros": 1531292766012047, "job": 800, "event": "compaction_finished", "compaction_time_micros": 240906, "output_level": 1, "num_output_files": 1, "total_output_size": 17392766, "num_input_records": 3978, "num_output_records": 3468, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:06:06.012919 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292766012914, "job": 800, "event": "table_file_deletion", "file_number": 10326}
>2018-07-11 07:06:06.015929 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292766015927, "job": 800, "event": "table_file_deletion", "file_number": 10324}
>2018-07-11 07:06:15.366923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 42/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:15.522942 7f29149e4700  1 mon.magna102@0(leader).log v19959 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19959)
>2018-07-11 07:06:25.371332 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 41/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:25.523778 7f29149e4700  1 mon.magna102@0(leader).log v19960 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19960)
>2018-07-11 07:06:35.375921 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 40/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:35.533034 7f29149e4700  1 mon.magna102@0(leader).log v19961 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19961)
>2018-07-11 07:06:45.380366 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 39/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:45.533844 7f29149e4700  1 mon.magna102@0(leader).log v19962 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19962)
>2018-07-11 07:06:52.501260 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10328. Immutable memtables: 0.
>
>2018-07-11 07:06:52.501302 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 801] Syncing log #10325
>2018-07-11 07:06:52.517796 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.501293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:06:52.517808 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 801] Flushing memtable with next log file: 10328
>
>2018-07-11 07:06:52.517854 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812517826, "job": 801, "event": "flush_started", "num_memtables": 1, "num_entries": 448, "num_deletes": 250, "memory_usage": 460472}
>2018-07-11 07:06:52.517857 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 801] Level-0 flush table #10329: started
>2018-07-11 07:06:52.560005 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812559991, "cf_name": "default", "job": 801, "event": "table_file_creation", "file_number": 10329, "file_size": 320074, "table_properties": {"data_size": 316300, "index_size": 1247, "filter_size": 1608, "raw_key_size": 6565, "raw_average_key_size": 21, "raw_value_size": 311321, "raw_average_value_size": 1037, "num_data_blocks": 39, "num_entries": 300, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:06:52.560015 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 801] Level-0 flush table #10329: 320074 bytes OK
>2018-07-11 07:06:52.618356 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.576544) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10329 started
>2018-07-11 07:06:52.618364 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.618321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10329: memtable #1 done
>2018-07-11 07:06:52.618366 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.618335) EVENT_LOG_v1 {"time_micros": 1531292812618329, "job": 801, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:06:52.618368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.618346) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:06:52.618387 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 801] Try to delete WAL files size 450108, prev total WAL file size 450108, number of live WAL files 2.
>
>2018-07-11 07:06:52.618565 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:06:52.618614 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.618608) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:06:52.618624 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 802] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:06:52.618630 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 801 Base level 0, inputs: [10329(312KB)], [10327(16MB)]
>
>2018-07-11 07:06:52.618643 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812618633, "job": 802, "event": "compaction_started", "files_L0": [10329], "files_L1": [10327], "score": -1, "input_data_size": 17712840}
>2018-07-11 07:06:52.802388 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 802] Generated table #10330: 3277 keys, 16122081 bytes
>2018-07-11 07:06:52.802439 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812802410, "cf_name": "default", "job": 802, "event": "table_file_creation", "file_number": 10330, "file_size": 16122081, "table_properties": {"data_size": 16004303, "index_size": 65955, "filter_size": 50896, "raw_key_size": 65980, "raw_average_key_size": 20, "raw_value_size": 15911353, "raw_average_value_size": 4855, "num_data_blocks": 1925, "num_entries": 3277, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:06:52.843904 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 802] Compacted 1@0 + 1@1 files to L1 => 16122081 bytes
>2018-07-11 07:06:52.885782 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.885694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.6 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.6) out(15.4), read-write-amplify(105.7) write-amplify(50.4) OK, records in: 3768, records dropped: 491
>
>2018-07-11 07:06:52.885792 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:06:52.885716) EVENT_LOG_v1 {"time_micros": 1531292812885706, "job": 802, "event": "compaction_finished", "compaction_time_micros": 225244, "output_level": 1, "num_output_files": 1, "total_output_size": 16122081, "num_input_records": 3768, "num_output_records": 3277, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:06:52.885938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812885933, "job": 802, "event": "table_file_deletion", "file_number": 10329}
>2018-07-11 07:06:52.889059 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531292812889057, "job": 802, "event": "table_file_deletion", "file_number": 10327}
>2018-07-11 07:06:52.889193 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:06:55.385109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 38/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:06:55.553310 7f29149e4700  1 mon.magna102@0(leader).log v19963 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19963)
>2018-07-11 07:07:01.387603 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 37/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:01.428560 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:07:01.562199 7f29149e4700  1 mon.magna102@0(leader).log v19964 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19964)
>2018-07-11 07:07:11.392167 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 36/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:11.539427 7f29149e4700  1 mon.magna102@0(leader).log v19965 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19965)
>2018-07-11 07:07:21.396565 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 35/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:21.548512 7f29149e4700  1 mon.magna102@0(leader).log v19966 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19966)
>2018-07-11 07:07:31.400996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 34/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:31.549402 7f29149e4700  1 mon.magna102@0(leader).log v19967 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19967)
>2018-07-11 07:07:41.405416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 33/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:41.558671 7f29149e4700  1 mon.magna102@0(leader).log v19968 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19968)
>2018-07-11 07:07:51.409998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 32/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:07:51.559501 7f29149e4700  1 mon.magna102@0(leader).log v19969 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19969)
>2018-07-11 07:08:01.414412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 31/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:01.451708 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:08:01.568686 7f29149e4700  1 mon.magna102@0(leader).log v19970 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19970)
>2018-07-11 07:08:03.833668 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:08:03.833692 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3027676421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:08:04.543978 7f29149e4700  1 mon.magna102@0(leader).log v19971 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19971)
>2018-07-11 07:08:09.933835 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:08:09.933858 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3975090070' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:08:10.562491 7f29149e4700  1 mon.magna102@0(leader).log v19972 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19972)
>2018-07-11 07:08:11.562504 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 30/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:11.679367 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:08:11.679395 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2171866494' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:08:11.712848 7f29149e4700  1 mon.magna102@0(leader).log v19973 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19973)
>2018-07-11 07:08:12.779683 7f29149e4700  1 mon.magna102@0(leader).log v19974 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 19974)
>2018-07-11 07:08:17.462479 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 29/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:17.580380 7f29149e4700  1 mon.magna102@0(leader).log v19975 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19975)
>2018-07-11 07:08:25.425783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 28/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:25.589650 7f29149e4700  1 mon.magna102@0(leader).log v19976 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19976)
>2018-07-11 07:08:35.430255 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 27/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:35.599011 7f29149e4700  1 mon.magna102@0(leader).log v19977 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19977)
>2018-07-11 07:08:45.434579 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 26/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:45.608216 7f29149e4700  1 mon.magna102@0(leader).log v19978 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19978)
>2018-07-11 07:08:55.439207 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 25/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:08:55.592350 7f29149e4700  1 mon.magna102@0(leader).log v19979 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19979)
>2018-07-11 07:09:01.475924 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:09:05.443580 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 24/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:05.594430 7f29149e4700  1 mon.magna102@0(leader).log v19980 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19980)
>2018-07-11 07:09:15.448092 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 23/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:15.595371 7f29149e4700  1 mon.magna102@0(leader).log v19981 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19981)
>2018-07-11 07:09:21.450625 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 22/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:21.604282 7f29149e4700  1 mon.magna102@0(leader).log v19982 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19982)
>2018-07-11 07:09:31.455121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 21/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:31.605088 7f29149e4700  1 mon.magna102@0(leader).log v19983 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19983)
>2018-07-11 07:09:41.459540 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 20/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:41.614353 7f29149e4700  1 mon.magna102@0(leader).log v19984 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19984)
>2018-07-11 07:09:51.464110 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 19/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:09:51.615168 7f29149e4700  1 mon.magna102@0(leader).log v19985 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19985)
>2018-07-11 07:10:01.468877 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 18/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:01.507584 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:10:01.624662 7f29149e4700  1 mon.magna102@0(leader).log v19986 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19986)
>2018-07-11 07:10:11.473471 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 17/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:11.643263 7f29149e4700  1 mon.magna102@0(leader).log v19987 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19987)
>2018-07-11 07:10:21.477986 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 16/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:21.652458 7f29149e4700  1 mon.magna102@0(leader).log v19988 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19988)
>2018-07-11 07:10:31.482394 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 15/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:31.653330 7f29149e4700  1 mon.magna102@0(leader).log v19989 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19989)
>2018-07-11 07:10:37.520378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 14/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:37.637406 7f29149e4700  1 mon.magna102@0(leader).log v19990 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19990)
>2018-07-11 07:10:45.488887 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 13/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:45.639264 7f29149e4700  1 mon.magna102@0(leader).log v19991 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19991)
>2018-07-11 07:10:55.493378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 12/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:10:55.640430 7f29149e4700  1 mon.magna102@0(leader).log v19992 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19992)
>2018-07-11 07:11:01.532313 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:11:05.497849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 11/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:05.649674 7f29149e4700  1 mon.magna102@0(leader).log v19993 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19993)
>2018-07-11 07:11:15.502400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 10/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:15.658874 7f29149e4700  1 mon.magna102@0(leader).log v19994 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19994)
>2018-07-11 07:11:25.506855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 9/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:25.651392 7f29149e4700  1 mon.magna102@0(leader).log v19995 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19995)
>2018-07-11 07:11:35.511339 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 8/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:35.668897 7f29149e4700  1 mon.magna102@0(leader).log v19996 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19996)
>2018-07-11 07:11:41.513887 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 7/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:41.669490 7f29149e4700  1 mon.magna102@0(leader).log v19997 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19997)
>2018-07-11 07:11:51.518322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 6/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:11:51.688372 7f29149e4700  1 mon.magna102@0(leader).log v19998 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19998)
>2018-07-11 07:12:01.522826 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 5/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:12:01.563911 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:12:01.697604 7f29149e4700  1 mon.magna102@0(leader).log v19999 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 19999)
>2018-07-11 07:12:11.527258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 4/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:12:11.673450 7f29149e4700  1 mon.magna102@0(leader).log v20000 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20000)
>2018-07-11 07:12:21.531712 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 3/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:12:21.682661 7f29149e4700  1 mon.magna102@0(leader).log v20001 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20001)
>2018-07-11 07:12:31.536189 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:12:31.683516 7f29149e4700  1 mon.magna102@0(leader).log v20002 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20002)
>2018-07-11 07:12:41.540645 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 07:12:41.692707 7f29149e4700  1 mon.magna102@0(leader).log v20003 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20003)
>2018-07-11 07:12:45.542490 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/24217500 objects degraded (0.000%), 1 pg degraded)
>2018-07-11 07:12:45.718266 7f29149e4700  1 mon.magna102@0(leader).log v20004 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20004)
>2018-07-11 07:12:49.494544 7f29149e4700  1 mon.magna102@0(leader).log v20005 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20005)
>2018-07-11 07:12:49.678860 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10331. Immutable memtables: 0.
>
>2018-07-11 07:12:49.678921 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 803] Syncing log #10328
>2018-07-11 07:12:49.695232 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.678911) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:12:49.695244 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 803] Flushing memtable with next log file: 10331
>
>2018-07-11 07:12:49.695258 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293169695250, "job": 803, "event": "flush_started", "num_memtables": 1, "num_entries": 1853, "num_deletes": 251, "memory_usage": 3706768}
>2018-07-11 07:12:49.695261 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 803] Level-0 flush table #10332: started
>2018-07-11 07:12:49.762455 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293169762440, "cf_name": "default", "job": 803, "event": "table_file_creation", "file_number": 10332, "file_size": 3547062, "table_properties": {"data_size": 3519588, "index_size": 14921, "filter_size": 11629, "raw_key_size": 15388, "raw_average_key_size": 20, "raw_value_size": 3498531, "raw_average_value_size": 4683, "num_data_blocks": 448, "num_entries": 747, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:12:49.762479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 803] Level-0 flush table #10332: 3547062 bytes OK
>2018-07-11 07:12:49.812476 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.778974) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10332 started
>2018-07-11 07:12:49.812484 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.812436) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10332: memtable #1 done
>2018-07-11 07:12:49.812486 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.812451) EVENT_LOG_v1 {"time_micros": 1531293169812445, "job": 803, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:12:49.812502 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.812465) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:12:49.812508 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 803] Try to delete WAL files size 3669763, prev total WAL file size 3669763, number of live WAL files 2.
>
>2018-07-11 07:12:49.813381 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:12:49.813431 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:12:49.813424) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:12:49.813441 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 804] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:12:49.813447 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 803 Base level 0, inputs: [10332(3463KB)], [10330(15MB)]
>
>2018-07-11 07:12:49.813460 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293169813450, "job": 804, "event": "compaction_started", "files_L0": [10332], "files_L1": [10330], "score": -1, "input_data_size": 19669143}
>2018-07-11 07:12:50.021539 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 804] Generated table #10333: 3514 keys, 17586235 bytes
>2018-07-11 07:12:50.021588 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293170021560, "cf_name": "default", "job": 804, "event": "table_file_creation", "file_number": 10333, "file_size": 17586235, "table_properties": {"data_size": 17456844, "index_size": 72875, "filter_size": 55589, "raw_key_size": 71121, "raw_average_key_size": 20, "raw_value_size": 17355600, "raw_average_value_size": 4938, "num_data_blocks": 2121, "num_entries": 3514, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:12:50.054698 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 804] Compacted 1@0 + 1@1 files to L1 => 17586235 bytes
>2018-07-11 07:12:50.088244 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:12:50.088169) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 81.5 rd, 72.9 wr, level 1, files in(1, 1) out(1) MB in(3.4, 15.4) out(16.8), read-write-amplify(10.5) write-amplify(5.0) OK, records in: 4024, records dropped: 510
>
>2018-07-11 07:12:50.088253 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:12:50.088197) EVENT_LOG_v1 {"time_micros": 1531293170088187, "job": 804, "event": "compaction_finished", "compaction_time_micros": 241222, "output_level": 1, "num_output_files": 1, "total_output_size": 17586235, "num_input_records": 4024, "num_output_records": 3514, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:12:50.088988 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293170088984, "job": 804, "event": "table_file_deletion", "file_number": 10332}
>2018-07-11 07:12:50.092130 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293170092127, "job": 804, "event": "table_file_deletion", "file_number": 10330}
>2018-07-11 07:12:51.255465 7f29149e4700  1 mon.magna102@0(leader).log v20006 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20006)
>2018-07-11 07:13:01.589505 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 07:14:01.611673 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 07:15:01.642051 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 07:15:12.726624 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10334. Immutable memtables: 0.
>
>2018-07-11 07:15:12.726669 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 805] Syncing log #10331
>2018-07-11 07:15:12.743160 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.726659) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:15:12.743171 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 805] Flushing memtable with next log file: 10334
>
>2018-07-11 07:15:12.743191 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293312743177, "job": 805, "event": "flush_started", "num_memtables": 1, "num_entries": 760, "num_deletes": 250, "memory_usage": 1105952}
>2018-07-11 07:15:12.743195 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 805] Level-0 flush table #10335: started
>2018-07-11 07:15:12.785354 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293312785340, "cf_name": "default", "job": 805, "event": "table_file_creation", "file_number": 10335, "file_size": 610054, "table_properties": {"data_size": 604018, "index_size": 2578, "filter_size": 2539, "raw_key_size": 7342, "raw_average_key_size": 21, "raw_value_size": 597533, "raw_average_value_size": 1762, "num_data_blocks": 81, "num_entries": 339, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:15:12.785364 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 805] Level-0 flush table #10335: 610054 bytes OK
>2018-07-11 07:15:12.835386 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.801898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10335 started
>2018-07-11 07:15:12.835394 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.835350) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10335: memtable #1 done
>2018-07-11 07:15:12.835397 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.835364) EVENT_LOG_v1 {"time_micros": 1531293312835358, "job": 805, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:15:12.835399 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.835376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:15:12.835405 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 805] Try to delete WAL files size 1090039, prev total WAL file size 1090039, number of live WAL files 2.
>
>2018-07-11 07:15:12.835688 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:15:12.835739 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:15:12.835732) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:15:12.835749 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 806] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:15:12.835755 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 805 Base level 0, inputs: [10335(595KB)], [10333(16MB)]
>
>2018-07-11 07:15:12.835767 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293312835758, "job": 806, "event": "compaction_started", "files_L0": [10335], "files_L1": [10333], "score": -1, "input_data_size": 18196289}
>2018-07-11 07:15:13.036107 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 806] Generated table #10336: 3410 keys, 16931845 bytes
>2018-07-11 07:15:13.036156 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293313036129, "cf_name": "default", "job": 806, "event": "table_file_creation", "file_number": 10336, "file_size": 16931845, "table_properties": {"data_size": 16808468, "index_size": 69067, "filter_size": 53383, "raw_key_size": 68684, "raw_average_key_size": 20, "raw_value_size": 16711452, "raw_average_value_size": 4900, "num_data_blocks": 2016, "num_entries": 3410, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:15:13.069256 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 806] Compacted 1@0 + 1@1 files to L1 => 16931845 bytes
>2018-07-11 07:15:13.102816 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:15:13.102750) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.9 rd, 72.5 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.8) out(16.1), read-write-amplify(57.6) write-amplify(27.8) OK, records in: 3853, records dropped: 443
>
>2018-07-11 07:15:13.102825 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:15:13.102772) EVENT_LOG_v1 {"time_micros": 1531293313102762, "job": 806, "event": "compaction_finished", "compaction_time_micros": 233474, "output_level": 1, "num_output_files": 1, "total_output_size": 16931845, "num_input_records": 3853, "num_output_records": 3410, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:15:13.102994 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293313102990, "job": 806, "event": "table_file_deletion", "file_number": 10335}
>2018-07-11 07:15:13.106098 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293313106095, "job": 806, "event": "table_file_deletion", "file_number": 10333}
>2018-07-11 07:15:13.106153 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:16:01.657760 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:16:04.622464 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd dump"} v 0) v1
>2018-07-11 07:16:04.622491 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3663326878' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch
>2018-07-11 07:16:04.750033 7f29149e4700  1 mon.magna102@0(leader).log v20007 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20007)
>2018-07-11 07:16:10.379419 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:16:10.379443 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1138797478' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:16:10.767330 7f29149e4700  1 mon.magna102@0(leader).log v20008 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20008)
>2018-07-11 07:17:01.688688 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:18:01.717864 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:19:01.740107 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:20:01.770949 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:20:59.910082 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10337. Immutable memtables: 0.
>
>2018-07-11 07:20:59.910125 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 807] Syncing log #10334
>2018-07-11 07:20:59.926680 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:20:59.910116) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:20:59.926691 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 807] Flushing memtable with next log file: 10337
>
>2018-07-11 07:20:59.926718 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293659926709, "job": 807, "event": "flush_started", "num_memtables": 1, "num_entries": 1497, "num_deletes": 251, "memory_usage": 2673120}
>2018-07-11 07:20:59.926721 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 807] Level-0 flush table #10338: started
>2018-07-11 07:20:59.993883 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293659993868, "cf_name": "default", "job": 807, "event": "table_file_creation", "file_number": 10338, "file_size": 2551955, "table_properties": {"data_size": 2530577, "index_size": 11682, "filter_size": 8772, "raw_key_size": 12767, "raw_average_key_size": 20, "raw_value_size": 2513566, "raw_average_value_size": 4067, "num_data_blocks": 354, "num_entries": 618, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:20:59.993894 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 807] Level-0 flush table #10338: 2551955 bytes OK
>2018-07-11 07:21:00.043906 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.010400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10338 started
>2018-07-11 07:21:00.043914 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.043870) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10338: memtable #1 done
>2018-07-11 07:21:00.043917 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.043884) EVENT_LOG_v1 {"time_micros": 1531293660043878, "job": 807, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:21:00.043932 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.043896) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:21:00.043938 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 807] Try to delete WAL files size 2643233, prev total WAL file size 2643233, number of live WAL files 2.
>
>2018-07-11 07:21:00.044653 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:21:00.044846 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 07:21:00.044853 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 55686.9 total, 894.3 interval
>Cumulative writes: 77K writes, 369K keys, 77K commit groups, 1.0 writes per commit group, ingest: 0.61 GB, 0.01 MB/s
>Cumulative WAL: 77K writes, 77K syncs, 1.00 writes per sync, written: 0.61 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4558 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.49 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.43 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.6        33       404    0.082       0      0
>  L1      1/1   16.15 MB   0.0      6.3     0.5      5.8       5.8      0.0       0.0  10.7     72.4     66.2        89       402    0.222   1427K   170K
> Sum      2/2   18.58 MB   0.0      6.3     0.5      5.8       6.3      0.5       0.0  11.7     52.8     52.8       122       806    0.152   1427K   170K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.7     58.4     58.4         1         8    0.153     15K   1954
>Uptime(secs): 55686.9 total, 55686.9 interval
>Flush(GB): cumulative 0.540, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.31 GB write, 0.12 MB/s write, 6.31 GB read, 0.12 MB/s read, 122.5 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.43 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.6        33       404    0.082       0      0
>  L1      1/1   16.15 MB   0.0      6.3     0.5      5.8       5.8      0.0       0.0  10.7     72.4     66.2        89       402    0.222   1427K   170K
> Sum      2/2   18.58 MB   0.0      6.3     0.5      5.8       6.3      0.5       0.0  11.7     52.8     52.8       122       806    0.152   1427K   170K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 55686.9 total, 55686.9 interval
>Flush(GB): cumulative 0.540, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.31 GB write, 0.12 MB/s write, 6.31 GB read, 0.12 MB/s read, 122.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 07:21:00.044874 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.044868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:21:00.044879 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 808] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:21:00.044884 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 807 Base level 0, inputs: [10338(2492KB)], [10336(16MB)]
>
>2018-07-11 07:21:00.044897 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293660044888, "job": 808, "event": "compaction_started", "files_L0": [10338], "files_L1": [10336], "score": -1, "input_data_size": 19483800}
>2018-07-11 07:21:00.252947 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 808] Generated table #10339: 3516 keys, 17404164 bytes
>2018-07-11 07:21:00.252991 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293660252964, "cf_name": "default", "job": 808, "event": "table_file_creation", "file_number": 10339, "file_size": 17404164, "table_properties": {"data_size": 17275201, "index_size": 72809, "filter_size": 55227, "raw_key_size": 71150, "raw_average_key_size": 20, "raw_value_size": 17173981, "raw_average_value_size": 4884, "num_data_blocks": 2119, "num_entries": 3516, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:21:00.286124 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 808] Compacted 1@0 + 1@1 files to L1 => 17404164 bytes
>2018-07-11 07:21:00.319662 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.319595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.8 rd, 72.2 wr, level 1, files in(1, 1) out(1) MB in(2.4, 16.1) out(16.6), read-write-amplify(14.5) write-amplify(6.8) OK, records in: 4028, records dropped: 512
>
>2018-07-11 07:21:00.319671 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:21:00.319616) EVENT_LOG_v1 {"time_micros": 1531293660319607, "job": 808, "event": "compaction_finished", "compaction_time_micros": 241209, "output_level": 1, "num_output_files": 1, "total_output_size": 17404164, "num_input_records": 4028, "num_output_records": 3516, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:21:00.320279 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293660320274, "job": 808, "event": "table_file_deletion", "file_number": 10338}
>2018-07-11 07:21:00.323621 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293660323619, "job": 808, "event": "table_file_deletion", "file_number": 10336}
>2018-07-11 07:21:01.795048 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:21:57.072810 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush dump"} v 0) v1
>2018-07-11 07:21:57.072838 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3297836731' entity='client.admin' cmd=[{"prefix": "osd crush dump"}]: dispatch
>2018-07-11 07:21:57.192130 7f29149e4700  1 mon.magna102@0(leader).log v20009 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20009)
>2018-07-11 07:22:01.817468 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:23:01.847782 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:23:32.925407 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10340. Immutable memtables: 0.
>
>2018-07-11 07:23:32.925450 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 809] Syncing log #10337
>2018-07-11 07:23:32.941936 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:23:32.925440) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:23:32.941947 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 809] Flushing memtable with next log file: 10340
>
>2018-07-11 07:23:32.941973 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293812941966, "job": 809, "event": "flush_started", "num_memtables": 1, "num_entries": 795, "num_deletes": 250, "memory_usage": 1181080}
>2018-07-11 07:23:32.941976 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 809] Level-0 flush table #10341: started
>2018-07-11 07:23:32.984151 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293812984137, "cf_name": "default", "job": 809, "event": "table_file_creation", "file_number": 10341, "file_size": 647495, "table_properties": {"data_size": 641179, "index_size": 2737, "filter_size": 2660, "raw_key_size": 7442, "raw_average_key_size": 21, "raw_value_size": 634510, "raw_average_value_size": 1844, "num_data_blocks": 86, "num_entries": 344, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:23:32.984162 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 809] Level-0 flush table #10341: 647495 bytes OK
>2018-07-11 07:23:33.034188 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.000663) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10341 started
>2018-07-11 07:23:33.034196 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.034145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10341: memtable #1 done
>2018-07-11 07:23:33.034200 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.034160) EVENT_LOG_v1 {"time_micros": 1531293813034154, "job": 809, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:23:33.034214 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.034171) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:23:33.034221 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 809] Try to delete WAL files size 1164263, prev total WAL file size 1164263, number of live WAL files 2.
>
>2018-07-11 07:23:33.034638 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:23:33.034700 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.034692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:23:33.034706 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 810] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:23:33.034712 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 809 Base level 0, inputs: [10341(632KB)], [10339(16MB)]
>
>2018-07-11 07:23:33.034725 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293813034715, "job": 810, "event": "compaction_started", "files_L0": [10341], "files_L1": [10339], "score": -1, "input_data_size": 18051659}
>2018-07-11 07:23:33.234892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 810] Generated table #10342: 3422 keys, 16817697 bytes
>2018-07-11 07:23:33.234938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293813234911, "cf_name": "default", "job": 810, "event": "table_file_creation", "file_number": 10342, "file_size": 16817697, "table_properties": {"data_size": 16694193, "index_size": 69339, "filter_size": 53238, "raw_key_size": 68923, "raw_average_key_size": 20, "raw_value_size": 16596820, "raw_average_value_size": 4850, "num_data_blocks": 2024, "num_entries": 3422, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:23:33.268065 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 810] Compacted 1@0 + 1@1 files to L1 => 16817697 bytes
>2018-07-11 07:23:33.301595 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.301528) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.4 rd, 72.1 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.6) out(16.0), read-write-amplify(53.9) write-amplify(26.0) OK, records in: 3860, records dropped: 438
>
>2018-07-11 07:23:33.301604 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:23:33.301550) EVENT_LOG_v1 {"time_micros": 1531293813301540, "job": 810, "event": "compaction_finished", "compaction_time_micros": 233326, "output_level": 1, "num_output_files": 1, "total_output_size": 16817697, "num_input_records": 3860, "num_output_records": 3422, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:23:33.301815 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293813301811, "job": 810, "event": "table_file_deletion", "file_number": 10341}
>2018-07-11 07:23:33.304938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293813304935, "job": 810, "event": "table_file_deletion", "file_number": 10339}
>2018-07-11 07:23:33.305055 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:23:59.480238 7f29149e4700  1 mon.magna102@0(leader).log v20010 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20010)
>2018-07-11 07:24:01.872187 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:25:01.902407 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:26:01.923242 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:26:31.243299 7f29149e4700  1 mon.magna102@0(leader).log v20011 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20011)
>2018-07-11 07:26:37.618869 7f29149e4700  1 mon.magna102@0(leader).log v20012 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20012)
>2018-07-11 07:26:38.752560 7f29149e4700  1 mon.magna102@0(leader).log v20013 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20013)
>2018-07-11 07:27:01.954172 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:28:01.976168 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:29:02.006504 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:29:08.132574 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10343. Immutable memtables: 0.
>
>2018-07-11 07:29:08.132619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 811] Syncing log #10340
>2018-07-11 07:29:08.149143 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.132609) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:29:08.149154 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 811] Flushing memtable with next log file: 10343
>
>2018-07-11 07:29:08.149188 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148149172, "job": 811, "event": "flush_started", "num_memtables": 1, "num_entries": 1464, "num_deletes": 251, "memory_usage": 2623200}
>2018-07-11 07:29:08.149191 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 811] Level-0 flush table #10344: started
>2018-07-11 07:29:08.208021 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148208006, "cf_name": "default", "job": 811, "event": "table_file_creation", "file_number": 10344, "file_size": 2505597, "table_properties": {"data_size": 2484648, "index_size": 11416, "filter_size": 8609, "raw_key_size": 12547, "raw_average_key_size": 20, "raw_value_size": 2468006, "raw_average_value_size": 4059, "num_data_blocks": 346, "num_entries": 608, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:29:08.208032 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 811] Level-0 flush table #10344: 2505597 bytes OK
>2018-07-11 07:29:08.258035 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.224454) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10344 started
>2018-07-11 07:29:08.258043 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.257997) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10344: memtable #1 done
>2018-07-11 07:29:08.258045 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.258012) EVENT_LOG_v1 {"time_micros": 1531294148258006, "job": 811, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:29:08.258047 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.258024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:29:08.258066 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 811] Try to delete WAL files size 2593573, prev total WAL file size 2593573, number of live WAL files 2.
>
>2018-07-11 07:29:08.258735 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:29:08.258798 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.258791) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:29:08.258805 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 812] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:29:08.258811 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 811 Base level 0, inputs: [10344(2446KB)], [10342(16MB)]
>
>2018-07-11 07:29:08.258824 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148258814, "job": 812, "event": "compaction_started", "files_L0": [10344], "files_L1": [10342], "score": -1, "input_data_size": 19323294}
>2018-07-11 07:29:08.458754 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 812] Generated table #10345: 3520 keys, 17234077 bytes
>2018-07-11 07:29:08.458798 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148458771, "cf_name": "default", "job": 812, "event": "table_file_creation", "file_number": 10345, "file_size": 17234077, "table_properties": {"data_size": 17105473, "index_size": 72779, "filter_size": 54898, "raw_key_size": 71223, "raw_average_key_size": 20, "raw_value_size": 17004212, "raw_average_value_size": 4830, "num_data_blocks": 2118, "num_entries": 3520, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:29:08.500291 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 812] Compacted 1@0 + 1@1 files to L1 => 17234077 bytes
>2018-07-11 07:29:08.542134 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.542066) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.0 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(2.4, 16.0) out(16.4), read-write-amplify(14.6) write-amplify(6.9) OK, records in: 4030, records dropped: 510
>
>2018-07-11 07:29:08.542146 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:29:08.542087) EVENT_LOG_v1 {"time_micros": 1531294148542078, "job": 812, "event": "compaction_finished", "compaction_time_micros": 241453, "output_level": 1, "num_output_files": 1, "total_output_size": 17234077, "num_input_records": 4030, "num_output_records": 3520, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:29:08.542716 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148542712, "job": 812, "event": "table_file_deletion", "file_number": 10344}
>2018-07-11 07:29:08.545989 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148545985, "job": 812, "event": "table_file_deletion", "file_number": 10342}
>2018-07-11 07:30:02.031122 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:31:02.053235 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:31:53.150167 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10346. Immutable memtables: 0.
>
>2018-07-11 07:31:53.150242 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 813] Syncing log #10343
>2018-07-11 07:31:53.166691 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.150233) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:31:53.166702 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 813] Flushing memtable with next log file: 10346
>
>2018-07-11 07:31:53.166728 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313166720, "job": 813, "event": "flush_started", "num_memtables": 1, "num_entries": 829, "num_deletes": 250, "memory_usage": 1249952}
>2018-07-11 07:31:53.166731 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 813] Level-0 flush table #10347: started
>2018-07-11 07:31:53.208873 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313208859, "cf_name": "default", "job": 813, "event": "table_file_creation", "file_number": 10347, "file_size": 671320, "table_properties": {"data_size": 664759, "index_size": 2898, "filter_size": 2744, "raw_key_size": 7450, "raw_average_key_size": 21, "raw_value_size": 657982, "raw_average_value_size": 1907, "num_data_blocks": 91, "num_entries": 345, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:31:53.208883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 813] Level-0 flush table #10347: 671320 bytes OK
>2018-07-11 07:31:53.258918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.225428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10347 started
>2018-07-11 07:31:53.258926 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.258882) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10347: memtable #1 done
>2018-07-11 07:31:53.258928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.258896) EVENT_LOG_v1 {"time_micros": 1531294313258890, "job": 813, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:31:53.258943 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.258908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:31:53.258950 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 813] Try to delete WAL files size 1232573, prev total WAL file size 1232573, number of live WAL files 2.
>
>2018-07-11 07:31:53.259332 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:31:53.259513 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 07:31:53.259519 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 56340.1 total, 653.2 interval
>Cumulative writes: 78K writes, 372K keys, 78K commit groups, 1.0 writes per commit group, ingest: 0.62 GB, 0.01 MB/s
>Cumulative WAL: 78K writes, 78K syncs, 1.00 writes per sync, written: 0.62 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 668 writes, 3088 keys, 668 commit groups, 1.0 writes per commit group, ingest: 4.76 MB, 0.01 MB/s
>Interval WAL: 668 writes, 668 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   655.59 KB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.7        33       407    0.082       0      0
>  L1      1/1   16.44 MB   0.0      6.4     0.5      5.8       5.8      0.0       0.0  10.7     72.4     66.2        90       405    0.222   1439K   171K
> Sum      2/2   17.08 MB   0.0      6.4     0.5      5.8       6.4      0.5       0.0  11.7     52.8     52.8       123       812    0.152   1439K   171K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0  14.5     59.7     58.0         1         6    0.151     11K   1460
>Uptime(secs): 56340.1 total, 56340.1 interval
>Flush(GB): cumulative 0.543, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.36 GB write, 0.12 MB/s write, 6.36 GB read, 0.12 MB/s read, 123.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   655.59 KB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.7        33       407    0.082       0      0
>  L1      1/1   16.44 MB   0.0      6.4     0.5      5.8       5.8      0.0       0.0  10.7     72.4     66.2        90       405    0.222   1439K   171K
> Sum      2/2   17.08 MB   0.0      6.4     0.5      5.8       6.4      0.5       0.0  11.7     52.8     52.8       123       812    0.152   1439K   171K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 56340.1 total, 56340.1 interval
>Flush(GB): cumulative 0.543, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.36 GB write, 0.12 MB/s write, 6.36 GB read, 0.12 MB/s read, 123.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 07:31:53.259543 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.259537) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:31:53.259548 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 814] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:31:53.259553 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 813 Base level 0, inputs: [10347(655KB)], [10345(16MB)]
>
>2018-07-11 07:31:53.259566 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313259557, "job": 814, "event": "compaction_started", "files_L0": [10347], "files_L1": [10345], "score": -1, "input_data_size": 17905397}
>2018-07-11 07:31:53.451320 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 814] Generated table #10348: 3435 keys, 16712105 bytes
>2018-07-11 07:31:53.451352 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313451338, "cf_name": "default", "job": 814, "event": "table_file_creation", "file_number": 10348, "file_size": 16712105, "table_properties": {"data_size": 16588411, "index_size": 69646, "filter_size": 53121, "raw_key_size": 69187, "raw_average_key_size": 20, "raw_value_size": 16490631, "raw_average_value_size": 4800, "num_data_blocks": 2033, "num_entries": 3435, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:31:53.484400 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 814] Compacted 1@0 + 1@1 files to L1 => 16712105 bytes
>2018-07-11 07:31:53.526268 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.526199) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.6 rd, 74.3 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.4) out(15.9), read-write-amplify(51.6) write-amplify(24.9) OK, records in: 3865, records dropped: 430
>
>2018-07-11 07:31:53.526277 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:31:53.526222) EVENT_LOG_v1 {"time_micros": 1531294313526212, "job": 814, "event": "compaction_finished", "compaction_time_micros": 224817, "output_level": 1, "num_output_files": 1, "total_output_size": 16712105, "num_input_records": 3865, "num_output_records": 3435, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:31:53.526467 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313526462, "job": 814, "event": "table_file_deletion", "file_number": 10347}
>2018-07-11 07:31:53.529623 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294313529607, "job": 814, "event": "table_file_deletion", "file_number": 10345}
>2018-07-11 07:31:53.529674 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:32:02.077320 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:33:02.107694 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:34:02.121419 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:35:02.158793 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:36:01.659580 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 07:36:01.659596 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 07:36:01.780987 7f29149e4700  1 mon.magna102@0(leader).log v20014 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20014)
>2018-07-11 07:36:02.158954 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:36:03.957476 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 07:36:03.957490 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 07:36:04.072993 7f29149e4700  1 mon.magna102@0(leader).log v20015 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20015)
>2018-07-11 07:36:04.668052 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:36:04.668066 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:36:04.904454 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 07:36:04.904470 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 07:36:04.943858 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:36:04.943873 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:36:04.956405 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:36:04.956419 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:36:05.139944 7f29149e4700  1 mon.magna102@0(leader).log v20016 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20016)
>2018-07-11 07:36:05.259314 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:36:05.259329 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:36:06.340468 7f29149e4700  1 mon.magna102@0(leader).log v20017 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20017)
>2018-07-11 07:36:06.343099 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 07:36:06.343112 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 07:36:06.557445 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 07:36:06.557459 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 07:36:06.611642 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 07:36:06.611656 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 07:36:07.407329 7f29149e4700  1 mon.magna102@0(leader).log v20018 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20018)
>2018-07-11 07:36:07.507463 7f29191ed700  1 mon.magna102@0(leader).osd e494 prepare_failure osd.10 10.8.128.107:6800/12665 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:36:07.507476 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/12665 reported failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:36:08.266455 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10349. Immutable memtables: 0.
>
>2018-07-11 07:36:08.266499 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 815] Syncing log #10346
>2018-07-11 07:36:08.282646 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.266489) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:36:08.282657 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 815] Flushing memtable with next log file: 10349
>
>2018-07-11 07:36:08.282684 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568282676, "job": 815, "event": "flush_started", "num_memtables": 1, "num_entries": 1440, "num_deletes": 506, "memory_usage": 2047952}
>2018-07-11 07:36:08.282687 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 815] Level-0 flush table #10350: started
>2018-07-11 07:36:08.342765 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568342751, "cf_name": "default", "job": 815, "event": "table_file_creation", "file_number": 10350, "file_size": 1951211, "table_properties": {"data_size": 1934035, "index_size": 8797, "filter_size": 7459, "raw_key_size": 16238, "raw_average_key_size": 20, "raw_value_size": 1917033, "raw_average_value_size": 2445, "num_data_blocks": 268, "num_entries": 784, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "506", "kMergeOperands": "0"}}
>2018-07-11 07:36:08.342776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 815] Level-0 flush table #10350: 1951211 bytes OK
>2018-07-11 07:36:08.392725 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.359289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10350 started
>2018-07-11 07:36:08.392733 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.392688) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10350: memtable #1 done
>2018-07-11 07:36:08.392736 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.392703) EVENT_LOG_v1 {"time_micros": 1531294568392697, "job": 815, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:36:08.392750 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.392715) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:36:08.392757 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 815] Try to delete WAL files size 2016957, prev total WAL file size 2016957, number of live WAL files 2.
>
>2018-07-11 07:36:08.393303 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:36:08.393353 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.393346) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 07:36:08.393363 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 816] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:36:08.393369 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 815 Base level 0, inputs: [10350(1905KB)], [10348(15MB)]
>
>2018-07-11 07:36:08.393383 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568393373, "job": 816, "event": "compaction_started", "files_L0": [10350], "files_L1": [10348], "score": -1, "input_data_size": 18663316}
>2018-07-11 07:36:08.610322 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 816] Generated table #10351: 3199 keys, 16177124 bytes
>2018-07-11 07:36:08.610371 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568610356, "cf_name": "default", "job": 816, "event": "table_file_creation", "file_number": 10351, "file_size": 16177124, "table_properties": {"data_size": 16055645, "index_size": 69007, "filter_size": 51545, "raw_key_size": 64473, "raw_average_key_size": 20, "raw_value_size": 15961647, "raw_average_value_size": 4989, "num_data_blocks": 2026, "num_entries": 3199, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:36:08.684196 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 816] Compacted 1@0 + 1@1 files to L1 => 16177124 bytes
>2018-07-11 07:36:08.759332 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.759259) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 64.2 rd, 55.6 wr, level 1, files in(1, 1) out(1) MB in(1.9, 15.9) out(15.4), read-write-amplify(17.9) write-amplify(8.3) OK, records in: 4219, records dropped: 1020
>
>2018-07-11 07:36:08.759341 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.759282) EVENT_LOG_v1 {"time_micros": 1531294568759272, "job": 816, "event": "compaction_finished", "compaction_time_micros": 290789, "output_level": 1, "num_output_files": 1, "total_output_size": 16177124, "num_input_records": 4219, "num_output_records": 3199, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:36:08.759839 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568759834, "job": 816, "event": "table_file_deletion", "file_number": 10350}
>2018-07-11 07:36:08.763199 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568763196, "job": 816, "event": "table_file_deletion", "file_number": 10348}
>2018-07-11 07:36:08.777396 7f29149e4700  1 mon.magna102@0(leader).log v20019 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20019)
>2018-07-11 07:36:08.777433 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10352. Immutable memtables: 0.
>
>2018-07-11 07:36:08.777473 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 817] Syncing log #10349
>2018-07-11 07:36:08.794286 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.777465) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:36:08.794298 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 817] Flushing memtable with next log file: 10352
>
>2018-07-11 07:36:08.794324 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568794317, "job": 817, "event": "flush_started", "num_memtables": 1, "num_entries": 15, "num_deletes": 0, "memory_usage": 37312}
>2018-07-11 07:36:08.794327 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 817] Level-0 flush table #10353: started
>2018-07-11 07:36:08.836410 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568836397, "cf_name": "default", "job": 817, "event": "table_file_creation", "file_number": 10353, "file_size": 36050, "table_properties": {"data_size": 34872, "index_size": 145, "filter_size": 118, "raw_key_size": 281, "raw_average_key_size": 23, "raw_value_size": 34514, "raw_average_value_size": 2876, "num_data_blocks": 5, "num_entries": 12, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:36:08.836420 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 817] Level-0 flush table #10353: 36050 bytes OK
>2018-07-11 07:36:08.886483 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.852988) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10353 started
>2018-07-11 07:36:08.886491 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.886449) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10353: memtable #1 done
>2018-07-11 07:36:08.886493 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.886463) EVENT_LOG_v1 {"time_micros": 1531294568886457, "job": 817, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:36:08.886495 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.886474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:36:08.886513 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 817] Try to delete WAL files size 34873, prev total WAL file size 34873, number of live WAL files 2.
>
>2018-07-11 07:36:08.886612 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:36:08.886659 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:08.886652) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 07:36:08.886668 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 818] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:36:08.886674 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 817 Base level 0, inputs: [10353(35KB)], [10351(15MB)]
>
>2018-07-11 07:36:08.886687 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294568886677, "job": 818, "event": "compaction_started", "files_L0": [10353], "files_L1": [10351], "score": -1, "input_data_size": 16213174}
>2018-07-11 07:36:09.070525 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 818] Generated table #10354: 3204 keys, 16211468 bytes
>2018-07-11 07:36:09.070575 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294569070547, "cf_name": "default", "job": 818, "event": "table_file_creation", "file_number": 10354, "file_size": 16211468, "table_properties": {"data_size": 16089746, "index_size": 69144, "filter_size": 51651, "raw_key_size": 64576, "raw_average_key_size": 20, "raw_value_size": 15995582, "raw_average_value_size": 4992, "num_data_blocks": 2030, "num_entries": 3204, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:36:09.103713 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 818] Compacted 1@0 + 1@1 files to L1 => 16211468 bytes
>2018-07-11 07:36:09.137248 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:09.137175) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.7 rd, 74.7 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.4) out(15.5), read-write-amplify(899.4) write-amplify(449.7) OK, records in: 3211, records dropped: 7
>
>2018-07-11 07:36:09.137257 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:09.137203) EVENT_LOG_v1 {"time_micros": 1531294569137194, "job": 818, "event": "compaction_finished", "compaction_time_micros": 217010, "output_level": 1, "num_output_files": 1, "total_output_size": 16211468, "num_input_records": 3211, "num_output_records": 3204, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:36:09.137356 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294569137351, "job": 818, "event": "table_file_deletion", "file_number": 10353}
>2018-07-11 07:36:09.140284 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294569140281, "job": 818, "event": "table_file_deletion", "file_number": 10351}
>2018-07-11 07:36:13.232858 7f291b9f2700  1 mon.magna102@0(leader).osd e494  we have enough reporters to mark osd.10 down
>2018-07-11 07:36:13.232885 7f291b9f2700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (6 reporters from different host after 25.725427 >= grace 21.618241)
>2018-07-11 07:36:13.233793 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 07:36:13.304995 7f29149e4700  1 mon.magna102@0(leader).osd e495 e495: 12 total, 11 up, 12 in
>2018-07-11 07:36:13.338942 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e495: 12 total, 11 up, 12 in
>2018-07-11 07:36:13.405217 7f29149e4700  1 mon.magna102@0(leader).log v20020 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20020)
>2018-07-11 07:36:13.488767 7f29149e4700  1 mon.magna102@0(leader).osd e496 e496: 12 total, 11 up, 12 in
>2018-07-11 07:36:13.522585 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e496: 12 total, 11 up, 12 in
>2018-07-11 07:36:14.472354 7f29149e4700  1 mon.magna102@0(leader).log v20021 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20021)
>2018-07-11 07:36:14.488716 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 15 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:36:14.488767 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 183573/24217500 objects degraded (0.758%), 6 pgs degraded (PG_DEGRADED)
>2018-07-11 07:36:14.622688 7f29149e4700  1 mon.magna102@0(leader).osd e497 e497: 12 total, 11 up, 12 in
>2018-07-11 07:36:14.656717 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e497: 12 total, 11 up, 12 in
>2018-07-11 07:36:15.539761 7f29149e4700  1 mon.magna102@0(leader).log v20022 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20022)
>2018-07-11 07:36:18.031707 7f29149e4700  1 mon.magna102@0(leader).log v20023 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20023)
>2018-07-11 07:36:20.154542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 64 pgs degraded (PG_DEGRADED)
>2018-07-11 07:36:20.154589 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 15 pgs peering)
>2018-07-11 07:36:20.307281 7f29149e4700  1 mon.magna102@0(leader).log v20024 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20024)
>2018-07-11 07:36:28.271870 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded (PG_DEGRADED)
>2018-07-11 07:36:29.291252 7f29149e4700  1 mon.magna102@0(leader).log v20025 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20025)
>2018-07-11 07:36:54.319943 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10355. Immutable memtables: 0.
>
>2018-07-11 07:36:54.319986 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 819] Syncing log #10352
>2018-07-11 07:36:54.336516 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.319977) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:36:54.336528 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 819] Flushing memtable with next log file: 10355
>
>2018-07-11 07:36:54.336554 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614336534, "job": 819, "event": "flush_started", "num_memtables": 1, "num_entries": 495, "num_deletes": 251, "memory_usage": 686080}
>2018-07-11 07:36:54.336557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 819] Level-0 flush table #10356: started
>2018-07-11 07:36:54.378702 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614378689, "cf_name": "default", "job": 819, "event": "table_file_creation", "file_number": 10356, "file_size": 599294, "table_properties": {"data_size": 593751, "index_size": 2208, "filter_size": 2416, "raw_key_size": 6870, "raw_average_key_size": 20, "raw_value_size": 587535, "raw_average_value_size": 1743, "num_data_blocks": 68, "num_entries": 337, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:36:54.378712 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 819] Level-0 flush table #10356: 599294 bytes OK
>2018-07-11 07:36:54.428753 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.395189) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10356 started
>2018-07-11 07:36:54.428761 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.428717) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10356: memtable #1 done
>2018-07-11 07:36:54.428763 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.428731) EVENT_LOG_v1 {"time_micros": 1531294614428725, "job": 819, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:36:54.428765 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.428743) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:36:54.428785 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 819] Try to delete WAL files size 674265, prev total WAL file size 674265, number of live WAL files 2.
>
>2018-07-11 07:36:54.429067 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:36:54.429116 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.429109) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:36:54.429125 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 820] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:36:54.429131 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 819 Base level 0, inputs: [10356(585KB)], [10354(15MB)]
>
>2018-07-11 07:36:54.429144 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614429135, "job": 820, "event": "compaction_started", "files_L0": [10356], "files_L1": [10354], "score": -1, "input_data_size": 16810762}
>2018-07-11 07:36:54.612797 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 820] Generated table #10357: 3027 keys, 14915326 bytes
>2018-07-11 07:36:54.612842 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614612814, "cf_name": "default", "job": 820, "event": "table_file_creation", "file_number": 10357, "file_size": 14915326, "table_properties": {"data_size": 14803670, "index_size": 63223, "filter_size": 47506, "raw_key_size": 61091, "raw_average_key_size": 20, "raw_value_size": 14716108, "raw_average_value_size": 4861, "num_data_blocks": 1848, "num_entries": 3027, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:36:54.645981 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 820] Compacted 1@0 + 1@1 files to L1 => 14915326 bytes
>2018-07-11 07:36:54.679509 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.679439) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.5 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(0.6, 15.5) out(14.2), read-write-amplify(52.9) write-amplify(24.9) OK, records in: 3541, records dropped: 514
>
>2018-07-11 07:36:54.679517 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:36:54.679462) EVENT_LOG_v1 {"time_micros": 1531294614679451, "job": 820, "event": "compaction_finished", "compaction_time_micros": 216821, "output_level": 1, "num_output_files": 1, "total_output_size": 14915326, "num_input_records": 3541, "num_output_records": 3027, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:36:54.679718 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614679713, "job": 820, "event": "table_file_deletion", "file_number": 10356}
>2018-07-11 07:36:54.682893 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294614682890, "job": 820, "event": "table_file_deletion", "file_number": 10354}
>2018-07-11 07:37:02.159137 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:37:16.183289 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded, 45 pgs undersized (PG_DEGRADED)
>2018-07-11 07:37:16.339969 7f29149e4700  1 mon.magna102@0(leader).log v20026 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20026)
>2018-07-11 07:37:23.273963 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded, 70 pgs undersized (PG_DEGRADED)
>2018-07-11 07:37:23.391786 7f29149e4700  1 mon.magna102@0(leader).log v20027 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20027)
>2018-07-11 07:38:02.159318 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:38:21.096598 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 07:38:21.096637 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/13927' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 07:38:21.097173 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 07:38:21.097219 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/13927' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 07:38:21.097309 7f29191ed700  0 mon.magna102@0(leader).osd e497 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 07:38:21.346953 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 07:38:21.388005 7f29149e4700  1 mon.magna102@0(leader).log v20028 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20028)
>2018-07-11 07:38:21.471663 7f29149e4700  1 mon.magna102@0(leader).osd e498 e498: 12 total, 12 up, 12 in
>2018-07-11 07:38:21.513575 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/13927 boot
>2018-07-11 07:38:21.513647 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e498: 12 total, 12 up, 12 in
>2018-07-11 07:38:21.513834 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 07:38:21.513871 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 07:38:21.655252 7f29149e4700  1 mon.magna102@0(leader).osd e499 e499: 12 total, 12 up, 12 in
>2018-07-11 07:38:21.688994 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e499: 12 total, 12 up, 12 in
>2018-07-11 07:38:22.455562 7f29149e4700  1 mon.magna102@0(leader).log v20029 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20029)
>2018-07-11 07:38:22.655191 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:38:22.655241 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1823081/24217500 objects degraded (7.528%), 60 pgs degraded, 63 pgs undersized (PG_DEGRADED)
>2018-07-11 07:38:22.790519 7f29149e4700  1 mon.magna102@0(leader).osd e500 e500: 12 total, 12 up, 12 in
>2018-07-11 07:38:22.824439 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e500: 12 total, 12 up, 12 in
>2018-07-11 07:38:22.999205 7f29149e4700  1 mon.magna102@0(leader).osd e501 e501: 12 total, 12 up, 12 in
>2018-07-11 07:38:23.032803 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e501: 12 total, 12 up, 12 in
>2018-07-11 07:38:23.524504 7f29149e4700  1 mon.magna102@0(leader).log v20030 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20030)
>2018-07-11 07:38:26.225213 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1120852/24217500 objects degraded (4.628%), 35 pgs degraded, 36 pgs undersized)
>2018-07-11 07:38:26.374981 7f29149e4700  1 mon.magna102@0(leader).log v20031 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20031)
>2018-07-11 07:38:28.225868 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs peering)
>2018-07-11 07:38:28.375276 7f29149e4700  1 mon.magna102@0(leader).log v20032 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20032)
>2018-07-11 07:39:02.159497 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:40:02.159643 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:40:07.608531 7f29149e4700  1 mon.magna102@0(leader).log v20033 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20033)
>2018-07-11 07:40:13.376062 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10358. Immutable memtables: 0.
>
>2018-07-11 07:40:13.376105 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 821] Syncing log #10355
>2018-07-11 07:40:13.392597 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.376096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:40:13.392609 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 821] Flushing memtable with next log file: 10358
>
>2018-07-11 07:40:13.392635 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813392627, "job": 821, "event": "flush_started", "num_memtables": 1, "num_entries": 1053, "num_deletes": 250, "memory_usage": 2221664}
>2018-07-11 07:40:13.392638 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 821] Level-0 flush table #10359: started
>2018-07-11 07:40:13.434779 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813434764, "cf_name": "default", "job": 821, "event": "table_file_creation", "file_number": 10359, "file_size": 1277554, "table_properties": {"data_size": 1268046, "index_size": 4257, "filter_size": 4331, "raw_key_size": 8704, "raw_average_key_size": 21, "raw_value_size": 1259365, "raw_average_value_size": 3109, "num_data_blocks": 133, "num_entries": 405, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:40:13.434789 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 821] Level-0 flush table #10359: 1277554 bytes OK
>2018-07-11 07:40:13.484844 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.451326) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10359 started
>2018-07-11 07:40:13.484853 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.484809) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10359: memtable #1 done
>2018-07-11 07:40:13.484855 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.484823) EVENT_LOG_v1 {"time_micros": 1531294813484817, "job": 821, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:40:13.484857 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.484835) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:40:13.484876 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 821] Try to delete WAL files size 2200440, prev total WAL file size 2200440, number of live WAL files 2.
>
>2018-07-11 07:40:13.485488 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:40:13.485553 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.485545) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:40:13.485563 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 822] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:40:13.485569 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 821 Base level 0, inputs: [10359(1247KB)], [10357(14MB)]
>
>2018-07-11 07:40:13.485583 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813485573, "job": 822, "event": "compaction_started", "files_L0": [10359], "files_L1": [10357], "score": -1, "input_data_size": 16192880}
>2018-07-11 07:40:13.668782 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 822] Generated table #10360: 3012 keys, 15109230 bytes
>2018-07-11 07:40:13.668832 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813668804, "cf_name": "default", "job": 822, "event": "table_file_creation", "file_number": 10360, "file_size": 15109230, "table_properties": {"data_size": 14998564, "index_size": 62071, "filter_size": 47668, "raw_key_size": 60501, "raw_average_key_size": 20, "raw_value_size": 14912110, "raw_average_value_size": 4950, "num_data_blocks": 1821, "num_entries": 3012, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:40:13.702098 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 822] Compacted 1@0 + 1@1 files to L1 => 15109230 bytes
>2018-07-11 07:40:13.735633 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.735559) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.8 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(1.2, 14.2) out(14.4), read-write-amplify(24.5) write-amplify(11.8) OK, records in: 3432, records dropped: 420
>
>2018-07-11 07:40:13.735642 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:40:13.735582) EVENT_LOG_v1 {"time_micros": 1531294813735572, "job": 822, "event": "compaction_finished", "compaction_time_micros": 216499, "output_level": 1, "num_output_files": 1, "total_output_size": 15109230, "num_input_records": 3432, "num_output_records": 3012, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:40:13.735973 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813735965, "job": 822, "event": "table_file_deletion", "file_number": 10359}
>2018-07-11 07:40:13.738725 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813738722, "job": 822, "event": "table_file_deletion", "file_number": 10357}
>2018-07-11 07:40:13.738842 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:40:14.511401 7f29149e4700  1 mon.magna102@0(leader).log v20034 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20034)
>2018-07-11 07:40:51.432120 7f29149e4700  1 mon.magna102@0(leader).log v20035 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20035)
>2018-07-11 07:41:02.159833 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:42:02.160009 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:43:02.160175 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:44:02.160357 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:44:44.527389 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10361. Immutable memtables: 0.
>
>2018-07-11 07:44:44.527434 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 823] Syncing log #10358
>2018-07-11 07:44:44.543947 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.527424) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:44:44.543958 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 823] Flushing memtable with next log file: 10361
>
>2018-07-11 07:44:44.543985 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084543977, "job": 823, "event": "flush_started", "num_memtables": 1, "num_entries": 1224, "num_deletes": 251, "memory_usage": 2094552}
>2018-07-11 07:44:44.543988 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 823] Level-0 flush table #10362: started
>2018-07-11 07:44:44.602825 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084602811, "cf_name": "default", "job": 823, "event": "table_file_creation", "file_number": 10362, "file_size": 1998802, "table_properties": {"data_size": 1981748, "index_size": 9126, "filter_size": 7008, "raw_key_size": 11081, "raw_average_key_size": 20, "raw_value_size": 1967714, "raw_average_value_size": 3657, "num_data_blocks": 278, "num_entries": 538, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:44:44.602836 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 823] Level-0 flush table #10362: 1998802 bytes OK
>2018-07-11 07:44:44.661206 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.619338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10362 started
>2018-07-11 07:44:44.661214 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.661163) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10362: memtable #1 done
>2018-07-11 07:44:44.661216 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.661177) EVENT_LOG_v1 {"time_micros": 1531295084661171, "job": 823, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:44:44.661218 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.661196) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:44:44.661238 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 823] Try to delete WAL files size 2069689, prev total WAL file size 2069689, number of live WAL files 2.
>
>2018-07-11 07:44:44.661803 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:44:44.661995 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 07:44:44.662001 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 57111.5 total, 771.4 interval
>Cumulative writes: 79K writes, 376K keys, 79K commit groups, 1.0 writes per commit group, ingest: 0.63 GB, 0.01 MB/s
>Cumulative WAL: 79K writes, 79K syncs, 1.00 writes per sync, written: 0.63 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 845 writes, 4227 keys, 845 commit groups, 1.0 writes per commit group, ingest: 6.67 MB, 0.01 MB/s
>Interval WAL: 845 writes, 845 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.91 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.7        34       412    0.082       0      0
>  L1      1/1   14.41 MB   0.0      6.4     0.5      5.9       5.9     -0.0       0.0  10.8     72.4     66.2        91       410    0.222   1457K   173K
> Sum      2/2   16.32 MB   0.0      6.4     0.5      5.9       6.4      0.5       0.0  11.7     52.8     52.8       125       822    0.152   1457K   173K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  14.5     54.8     54.2         1        10    0.149     18K   2391
>Uptime(secs): 57111.5 total, 57111.5 interval
>Flush(GB): cumulative 0.549, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.44 GB write, 0.12 MB/s write, 6.44 GB read, 0.12 MB/s read, 124.9 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.91 MB   0.0      0.0     0.0      0.0       0.5      0.5       0.0   1.0      0.0     16.7        34       412    0.082       0      0
>  L1      1/1   14.41 MB   0.0      6.4     0.5      5.9       5.9     -0.0       0.0  10.8     72.4     66.2        91       410    0.222   1457K   173K
> Sum      2/2   16.32 MB   0.0      6.4     0.5      5.9       6.4      0.5       0.0  11.7     52.8     52.8       125       822    0.152   1457K   173K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 57111.5 total, 57111.5 interval
>Flush(GB): cumulative 0.549, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.44 GB write, 0.12 MB/s write, 6.44 GB read, 0.12 MB/s read, 124.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 07:44:44.662022 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.662016) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:44:44.662028 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 824] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:44:44.662033 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 823 Base level 0, inputs: [10362(1951KB)], [10360(14MB)]
>
>2018-07-11 07:44:44.662046 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084662037, "job": 824, "event": "compaction_started", "files_L0": [10362], "files_L1": [10360], "score": -1, "input_data_size": 17108032}
>2018-07-11 07:44:44.845278 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 824] Generated table #10363: 3040 keys, 15200621 bytes
>2018-07-11 07:44:44.845323 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084845295, "cf_name": "default", "job": 824, "event": "table_file_creation", "file_number": 10363, "file_size": 15200621, "table_properties": {"data_size": 15088368, "index_size": 63236, "filter_size": 48090, "raw_key_size": 61335, "raw_average_key_size": 20, "raw_value_size": 15000625, "raw_average_value_size": 4934, "num_data_blocks": 1848, "num_entries": 3040, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:44:44.878444 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 824] Compacted 1@0 + 1@1 files to L1 => 15200621 bytes
>2018-07-11 07:44:44.911963 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.911898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.9, 14.4) out(14.5), read-write-amplify(16.2) write-amplify(7.6) OK, records in: 3550, records dropped: 510
>
>2018-07-11 07:44:44.911972 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:44:44.911919) EVENT_LOG_v1 {"time_micros": 1531295084911910, "job": 824, "event": "compaction_finished", "compaction_time_micros": 216382, "output_level": 1, "num_output_files": 1, "total_output_size": 15200621, "num_input_records": 3550, "num_output_records": 3040, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:44:44.912482 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084912478, "job": 824, "event": "table_file_deletion", "file_number": 10362}
>2018-07-11 07:44:44.915397 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295084915394, "job": 824, "event": "table_file_deletion", "file_number": 10360}
>2018-07-11 07:45:02.160528 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:45:18.032769 7f29149e4700  1 mon.magna102@0(leader).log v20036 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20036)
>2018-07-11 07:45:21.566571 7f29149e4700  1 mon.magna102@0(leader).log v20037 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20037)
>2018-07-11 07:45:38.359566 7f29149e4700  1 mon.magna102@0(leader).log v20038 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20038)
>2018-07-11 07:45:41.609988 7f29149e4700  1 mon.magna102@0(leader).log v20039 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20039)
>2018-07-11 07:45:58.345850 7f29149e4700  1 mon.magna102@0(leader).log v20040 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20040)
>2018-07-11 07:46:02.160695 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:47:02.160865 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:48:02.161034 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:48:33.592766 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10364. Immutable memtables: 0.
>
>2018-07-11 07:48:33.592810 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 825] Syncing log #10361
>2018-07-11 07:48:33.609269 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.592800) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:48:33.609281 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 825] Flushing memtable with next log file: 10364
>
>2018-07-11 07:48:33.609307 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313609299, "job": 825, "event": "flush_started", "num_memtables": 1, "num_entries": 1093, "num_deletes": 250, "memory_usage": 1813976}
>2018-07-11 07:48:33.609310 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 825] Level-0 flush table #10365: started
>2018-07-11 07:48:33.659785 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313659771, "cf_name": "default", "job": 825, "event": "table_file_creation", "file_number": 10365, "file_size": 996427, "table_properties": {"data_size": 987530, "index_size": 4217, "filter_size": 3760, "raw_key_size": 8446, "raw_average_key_size": 21, "raw_value_size": 979096, "raw_average_value_size": 2485, "num_data_blocks": 132, "num_entries": 394, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:48:33.659795 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 825] Level-0 flush table #10365: 996427 bytes OK
>2018-07-11 07:48:33.718156 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.676321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10365 started
>2018-07-11 07:48:33.718164 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.718119) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10365: memtable #1 done
>2018-07-11 07:48:33.718166 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.718134) EVENT_LOG_v1 {"time_micros": 1531295313718128, "job": 825, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:48:33.718189 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.718146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:48:33.718196 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 825] Try to delete WAL files size 1791769, prev total WAL file size 1791769, number of live WAL files 2.
>
>2018-07-11 07:48:33.718707 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:48:33.718768 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.718761) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:48:33.718775 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 826] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:48:33.718781 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 825 Base level 0, inputs: [10365(973KB)], [10363(14MB)]
>
>2018-07-11 07:48:33.718794 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313718784, "job": 826, "event": "compaction_started", "files_L0": [10365], "files_L1": [10363], "score": -1, "input_data_size": 16197048}
>2018-07-11 07:48:33.902233 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 826] Generated table #10366: 3034 keys, 15222467 bytes
>2018-07-11 07:48:33.902278 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313902250, "cf_name": "default", "job": 826, "event": "table_file_creation", "file_number": 10366, "file_size": 15222467, "table_properties": {"data_size": 15110882, "index_size": 62616, "filter_size": 48042, "raw_key_size": 60948, "raw_average_key_size": 20, "raw_value_size": 15023742, "raw_average_value_size": 4951, "num_data_blocks": 1837, "num_entries": 3034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:48:33.935335 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 826] Compacted 1@0 + 1@1 files to L1 => 15222467 bytes
>2018-07-11 07:48:33.968889 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.968818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.8 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.5) out(14.5), read-write-amplify(31.5) write-amplify(15.3) OK, records in: 3434, records dropped: 400
>
>2018-07-11 07:48:33.968898 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:48:33.968840) EVENT_LOG_v1 {"time_micros": 1531295313968831, "job": 826, "event": "compaction_finished", "compaction_time_micros": 216522, "output_level": 1, "num_output_files": 1, "total_output_size": 15222467, "num_input_records": 3434, "num_output_records": 3034, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:48:33.969197 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313969192, "job": 826, "event": "table_file_deletion", "file_number": 10365}
>2018-07-11 07:48:33.971923 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295313971920, "job": 826, "event": "table_file_deletion", "file_number": 10363}
>2018-07-11 07:48:33.972036 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:49:02.161222 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:50:02.161398 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:51:02.161587 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:52:02.161759 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:52:52.733156 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10367. Immutable memtables: 0.
>
>2018-07-11 07:52:52.733228 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 827] Syncing log #10364
>2018-07-11 07:52:52.749719 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.733218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:52:52.749731 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 827] Flushing memtable with next log file: 10367
>
>2018-07-11 07:52:52.749757 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295572749749, "job": 827, "event": "flush_started", "num_memtables": 1, "num_entries": 1166, "num_deletes": 251, "memory_usage": 1966336}
>2018-07-11 07:52:52.749761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 827] Level-0 flush table #10368: started
>2018-07-11 07:52:52.808599 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295572808584, "cf_name": "default", "job": 827, "event": "table_file_creation", "file_number": 10368, "file_size": 1875113, "table_properties": {"data_size": 1858971, "index_size": 8600, "filter_size": 6622, "raw_key_size": 10656, "raw_average_key_size": 20, "raw_value_size": 1845624, "raw_average_value_size": 3562, "num_data_blocks": 262, "num_entries": 518, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:52:52.808610 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 827] Level-0 flush table #10368: 1875113 bytes OK
>2018-07-11 07:52:52.859857 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.825110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10368 started
>2018-07-11 07:52:52.859865 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.859821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10368: memtable #1 done
>2018-07-11 07:52:52.859868 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.859835) EVENT_LOG_v1 {"time_micros": 1531295572859829, "job": 827, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:52:52.859870 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.859847) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:52:52.859889 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 827] Try to delete WAL files size 1942843, prev total WAL file size 1942843, number of live WAL files 2.
>
>2018-07-11 07:52:52.860409 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:52:52.860459 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:52:52.860452) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:52:52.860468 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 828] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:52:52.860475 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 827 Base level 0, inputs: [10368(1831KB)], [10366(14MB)]
>
>2018-07-11 07:52:52.860488 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295572860478, "job": 828, "event": "compaction_started", "files_L0": [10368], "files_L1": [10366], "score": -1, "input_data_size": 17097580}
>2018-07-11 07:52:53.043870 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 828] Generated table #10369: 3044 keys, 15084384 bytes
>2018-07-11 07:52:53.043919 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295573043892, "cf_name": "default", "job": 828, "event": "table_file_creation", "file_number": 10369, "file_size": 15084384, "table_properties": {"data_size": 14972392, "index_size": 63205, "filter_size": 47860, "raw_key_size": 61408, "raw_average_key_size": 20, "raw_value_size": 14884623, "raw_average_value_size": 4889, "num_data_blocks": 1847, "num_entries": 3044, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:52:53.077078 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 828] Compacted 1@0 + 1@1 files to L1 => 15084384 bytes
>2018-07-11 07:52:53.110607 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:52:53.110539) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.9 rd, 69.7 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.5) out(14.4), read-write-amplify(17.2) write-amplify(8.0) OK, records in: 3552, records dropped: 508
>
>2018-07-11 07:52:53.110616 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:52:53.110561) EVENT_LOG_v1 {"time_micros": 1531295573110551, "job": 828, "event": "compaction_finished", "compaction_time_micros": 216574, "output_level": 1, "num_output_files": 1, "total_output_size": 15084384, "num_input_records": 3552, "num_output_records": 3044, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:52:53.111070 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295573111066, "job": 828, "event": "table_file_deletion", "file_number": 10368}
>2018-07-11 07:52:53.114040 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295573114037, "job": 828, "event": "table_file_deletion", "file_number": 10366}
>2018-07-11 07:52:53.365576 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.365592 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.365621 7f29191ed700  1 mon.magna102@0(leader).osd e501  we're forcing failure of osd.10
>2018-07-11 07:52:53.365646 7f29191ed700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (connection refused reported by osd.6)
>2018-07-11 07:52:53.365789 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.365802 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.365878 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.365889 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.365971 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.365983 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.366063 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 07:52:53.366075 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 07:52:53.366155 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 07:52:53.366166 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 07:52:53.366262 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 07:52:53.366273 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 07:52:53.366354 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 07:52:53.366366 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 07:52:53.366445 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 07:52:53.366456 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 07:52:53.366532 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.366543 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.366615 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 07:52:53.366626 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 07:52:53.366699 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.366710 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.366780 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.366791 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.366859 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.366870 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.366941 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.366952 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.367024 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 07:52:53.367035 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 07:52:53.367104 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 07:52:53.367116 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 07:52:53.367193 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 07:52:53.367205 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 07:52:53.367277 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 07:52:53.367288 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 07:52:53.367357 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 07:52:53.367369 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 07:52:53.367438 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.367449 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.367522 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 07:52:53.367533 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 07:52:53.367613 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.367624 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.367696 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.367707 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.367779 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.367790 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.367861 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.367872 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.367941 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.367952 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.564848 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.564861 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.566000 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.566018 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.566109 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.566124 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.566245 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 07:52:53.566257 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 07:52:53.566363 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.566386 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.566530 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 07:52:53.566552 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 07:52:53.566682 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.566702 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.566792 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 07:52:53.566803 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 07:52:53.566877 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 07:52:53.566889 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 07:52:53.566959 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 07:52:53.566970 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 07:52:53.567038 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 07:52:53.567049 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 07:52:53.567120 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 07:52:53.567132 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 07:52:53.567213 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 07:52:53.567225 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 07:52:53.567303 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 07:52:53.567315 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 07:52:53.567385 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.567396 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.567465 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.567477 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.567548 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.567559 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.567633 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 07:52:53.567644 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 07:52:53.567713 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 07:52:53.567724 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 07:52:53.567795 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 07:52:53.567806 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 07:52:53.567877 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 07:52:53.567888 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 07:52:53.567957 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 07:52:53.567968 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 07:52:53.568035 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.568047 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.568118 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.568129 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.568206 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 07:52:53.568217 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 07:52:53.568289 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 07:52:53.568300 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 07:52:53.568373 7f29191ed700  1 mon.magna102@0(leader).osd e501 prepare_failure osd.10 10.8.128.107:6800/13927 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 07:52:53.568384 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/13927 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 07:52:53.770202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 07:52:53.803215 7f29149e4700  1 mon.magna102@0(leader).log v20041 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20041)
>2018-07-11 07:52:53.886428 7f29149e4700  1 mon.magna102@0(leader).osd e502 e502: 12 total, 11 up, 12 in
>2018-07-11 07:52:53.920036 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e502: 12 total, 11 up, 12 in
>2018-07-11 07:52:54.103326 7f29149e4700  1 mon.magna102@0(leader).osd e503 e503: 12 total, 11 up, 12 in
>2018-07-11 07:52:54.137490 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e503: 12 total, 11 up, 12 in
>2018-07-11 07:52:54.878761 7f29149e4700  1 mon.magna102@0(leader).log v20042 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20042)
>2018-07-11 07:52:55.103252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:52:55.103300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 134460/24217500 objects degraded (0.555%), 7 pgs degraded (PG_DEGRADED)
>2018-07-11 07:52:55.954099 7f29149e4700  1 mon.magna102@0(leader).log v20043 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20043)
>2018-07-11 07:52:59.754513 7f29149e4700  1 mon.magna102@0(leader).log v20044 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20044)
>2018-07-11 07:53:00.754575 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded (PG_DEGRADED)
>2018-07-11 07:53:00.754622 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering)
>2018-07-11 07:53:00.904930 7f29149e4700  1 mon.magna102@0(leader).log v20045 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20045)
>2018-07-11 07:53:02.161943 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:53:07.888770 7f29149e4700  1 mon.magna102@0(leader).log v20046 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20046)
>2018-07-11 07:53:10.022405 7f29149e4700  1 mon.magna102@0(leader).log v20047 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20047)
>2018-07-11 07:53:14.206169 7f29149e4700  1 mon.magna102@0(leader).log v20048 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20048)
>2018-07-11 07:53:28.619132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:53:28.799337 7f29149e4700  1 mon.magna102@0(leader).log v20049 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20049)
>2018-07-11 07:53:33.665864 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:53:33.834323 7f29149e4700  1 mon.magna102@0(leader).log v20050 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20050)
>2018-07-11 07:53:38.666123 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:53:38.784995 7f29149e4700  1 mon.magna102@0(leader).log v20051 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20051)
>2018-07-11 07:53:56.634324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded, 43 pgs undersized (PG_DEGRADED)
>2018-07-11 07:53:56.786394 7f29149e4700  1 mon.magna102@0(leader).log v20052 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20052)
>2018-07-11 07:54:02.162136 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:54:03.669777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2014227/24217500 objects degraded (8.317%), 65 pgs degraded, 70 pgs undersized (PG_DEGRADED)
>2018-07-11 07:54:03.803568 7f29149e4700  1 mon.magna102@0(leader).log v20053 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20053)
>2018-07-11 07:54:36.659274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 7 slow requests are blocked > 32 sec. Implicated osds 2,4 (REQUEST_SLOW)
>2018-07-11 07:54:36.814573 7f29149e4700  1 mon.magna102@0(leader).log v20054 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20054)
>2018-07-11 07:54:39.210639 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 07:54:39.210678 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14173' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 07:54:39.211283 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 07:54:39.211318 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14173' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 07:54:39.211408 7f29191ed700  0 mon.magna102@0(leader).osd e503 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 07:54:39.766815 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 07:54:39.799470 7f29149e4700  1 mon.magna102@0(leader).log v20055 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20055)
>2018-07-11 07:54:39.866482 7f29149e4700  1 mon.magna102@0(leader).osd e504 e504: 12 total, 12 up, 12 in
>2018-07-11 07:54:39.899922 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14173 boot
>2018-07-11 07:54:39.900000 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e504: 12 total, 12 up, 12 in
>2018-07-11 07:54:39.900173 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 07:54:39.900225 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 07:54:40.050050 7f29149e4700  1 mon.magna102@0(leader).osd e505 e505: 12 total, 12 up, 12 in
>2018-07-11 07:54:40.083908 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e505: 12 total, 12 up, 12 in
>2018-07-11 07:54:40.875302 7f29149e4700  1 mon.magna102@0(leader).log v20056 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20056)
>2018-07-11 07:54:41.184086 7f29149e4700  1 mon.magna102@0(leader).osd e506 e506: 12 total, 12 up, 12 in
>2018-07-11 07:54:41.217858 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e506: 12 total, 12 up, 12 in
>2018-07-11 07:54:42.275974 7f29149e4700  1 mon.magna102@0(leader).log v20057 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20057)
>2018-07-11 07:54:43.276041 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:54:43.276089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1823081/24217500 objects degraded (7.528%), 60 pgs degraded, 63 pgs undersized (PG_DEGRADED)
>2018-07-11 07:54:43.276107 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 07:54:43.434763 7f29149e4700  1 mon.magna102@0(leader).log v20058 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20058)
>2018-07-11 07:54:44.009960 7f29149e4700  1 mon.magna102@0(leader).osd e507 e507: 12 total, 12 up, 12 in
>2018-07-11 07:54:44.043649 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e507: 12 total, 12 up, 12 in
>2018-07-11 07:54:45.076923 7f29149e4700  1 mon.magna102@0(leader).osd e508 e508: 12 total, 12 up, 12 in
>2018-07-11 07:54:45.110626 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e508: 12 total, 12 up, 12 in
>2018-07-11 07:54:45.177291 7f29149e4700  1 mon.magna102@0(leader).log v20059 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20059)
>2018-07-11 07:54:46.243977 7f29149e4700  1 mon.magna102@0(leader).log v20060 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20060)
>2018-07-11 07:54:47.244031 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 07:54:47.244076 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 298758/24217500 objects degraded (1.234%), 12 pgs degraded, 13 pgs undersized)
>2018-07-11 07:54:47.394443 7f29149e4700  1 mon.magna102@0(leader).log v20061 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20061)
>2018-07-11 07:54:48.665162 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:54:48.811432 7f29149e4700  1 mon.magna102@0(leader).log v20062 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20062)
>2018-07-11 07:54:50.666129 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 6 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 07:54:50.820103 7f29149e4700  1 mon.magna102@0(leader).log v20063 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20063)
>2018-07-11 07:55:02.162330 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:55:02.671630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 07:55:02.821098 7f29149e4700  1 mon.magna102@0(leader).log v20064 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20064)
>2018-07-11 07:55:08.674524 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:08.830031 7f29149e4700  1 mon.magna102@0(leader).log v20065 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20065)
>2018-07-11 07:55:13.713295 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:13.830643 7f29149e4700  1 mon.magna102@0(leader).log v20066 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20066)
>2018-07-11 07:55:23.724005 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:23.849456 7f29149e4700  1 mon.magna102@0(leader).log v20067 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20067)
>2018-07-11 07:55:32.685888 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:32.858695 7f29149e4700  1 mon.magna102@0(leader).log v20068 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20068)
>2018-07-11 07:55:38.688781 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:38.834281 7f29149e4700  1 mon.magna102@0(leader).log v20069 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20069)
>2018-07-11 07:55:42.690603 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:55:42.843074 7f29149e4700  1 mon.magna102@0(leader).log v20070 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20070)
>2018-07-11 07:55:43.725265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 40 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:43.909907 7f29149e4700  1 mon.magna102@0(leader).log v20071 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20071)
>2018-07-11 07:55:48.735163 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 13 pgs inactive, 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:55:48.860584 7f29149e4700  1 mon.magna102@0(leader).log v20072 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20072)
>2018-07-11 07:55:52.695570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 44 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:52.864833 7f29149e4700  1 mon.magna102@0(leader).log v20073 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20073)
>2018-07-11 07:55:58.698633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 48 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:55:58.845057 7f29149e4700  1 mon.magna102@0(leader).log v20074 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20074)
>2018-07-11 07:56:02.162504 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8790 MB, avail 861 GB
>2018-07-11 07:56:03.735783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 52 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:56:03.837164 7f29149e4700  1 mon.magna102@0(leader).log v20075 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20075)
>2018-07-11 07:56:12.705586 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:56:12.854746 7f29149e4700  1 mon.magna102@0(leader).log v20076 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20076)
>2018-07-11 07:56:22.710534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 07:56:22.865164 7f29149e4700  1 mon.magna102@0(leader).log v20077 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20077)
>2018-07-11 07:56:28.713581 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs inactive, 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 07:56:28.857516 7f29149e4700  1 mon.magna102@0(leader).log v20078 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20078)
>2018-07-11 07:56:30.714603 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 4 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 07:56:30.866173 7f29149e4700  1 mon.magna102@0(leader).log v20079 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20079)
>2018-07-11 07:56:33.740644 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY)
>2018-07-11 07:56:33.849814 7f29149e4700  1 mon.magna102@0(leader).log v20080 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20080)
>2018-07-11 07:56:36.717282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 07:56:36.866991 7f29149e4700  1 mon.magna102@0(leader).log v20081 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20081)
>2018-07-11 07:56:43.758894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 12 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 07:56:43.859246 7f29149e4700  1 mon.magna102@0(leader).log v20082 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20082)
>2018-07-11 07:56:48.759522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 37 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 07:56:48.876440 7f29149e4700  1 mon.magna102@0(leader).log v20083 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20083)
>2018-07-11 07:56:53.478715 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:56:53.478736 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1787651070' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:56:53.759742 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 07:56:53.835613 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10370. Immutable memtables: 0.
>
>2018-07-11 07:56:53.868918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:53.835672) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:56:53.868931 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 829] Flushing memtable with next log file: 10370
>
>2018-07-11 07:56:53.868945 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295813868937, "job": 829, "event": "flush_started", "num_memtables": 1, "num_entries": 1506, "num_deletes": 250, "memory_usage": 3444272}
>2018-07-11 07:56:53.868949 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 829] Level-0 flush table #10371: started
>2018-07-11 07:56:53.934915 7f29149e4700  1 mon.magna102@0(leader).log v20084 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20084)
>2018-07-11 07:56:53.961106 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295813961092, "cf_name": "default", "job": 829, "event": "table_file_creation", "file_number": 10371, "file_size": 2257713, "table_properties": {"data_size": 2242049, "index_size": 7548, "filter_size": 7192, "raw_key_size": 11422, "raw_average_key_size": 21, "raw_value_size": 2229137, "raw_average_value_size": 4135, "num_data_blocks": 232, "num_entries": 539, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 07:56:53.961118 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 829] Level-0 flush table #10371: 2257713 bytes OK
>2018-07-11 07:56:54.019516 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:53.977739) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10371 started
>2018-07-11 07:56:54.019524 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.019479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10371: memtable #1 done
>2018-07-11 07:56:54.019526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.019494) EVENT_LOG_v1 {"time_micros": 1531295814019487, "job": 829, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:56:54.019541 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.019506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:56:54.019548 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 829] Try to delete WAL files size 3414422, prev total WAL file size 3433556, number of live WAL files 2.
>
>2018-07-11 07:56:54.020398 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:56:54.020578 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 07:56:54.020585 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 57840.9 total, 729.4 interval
>Cumulative writes: 79K writes, 380K keys, 79K commit groups, 1.0 writes per commit group, ingest: 0.63 GB, 0.01 MB/s
>Cumulative WAL: 79K writes, 79K syncs, 1.00 writes per sync, written: 0.63 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 851 writes, 3773 keys, 851 commit groups, 1.0 writes per commit group, ingest: 6.84 MB, 0.01 MB/s
>Interval WAL: 851 writes, 851 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        34       415    0.082       0      0
>  L1      1/1   14.39 MB   0.0      6.5     0.6      5.9       5.9     -0.0       0.0  10.8     72.4     66.2        92       413    0.222   1467K   175K
> Sum      2/2   16.54 MB   0.0      6.5     0.6      5.9       6.5      0.6       0.0  11.7     52.8     52.8       126       828    0.152   1467K   175K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.9     53.4     53.6         1         6    0.150     10K   1418
>Uptime(secs): 57840.9 total, 57840.9 interval
>Flush(GB): cumulative 0.553, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.49 GB write, 0.11 MB/s write, 6.49 GB read, 0.11 MB/s read, 125.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        34       415    0.082       0      0
>  L1      1/1   14.39 MB   0.0      6.5     0.6      5.9       5.9     -0.0       0.0  10.8     72.4     66.2        92       413    0.222   1467K   175K
> Sum      2/2   16.54 MB   0.0      6.5     0.6      5.9       6.5      0.6       0.0  11.7     52.8     52.8       126       828    0.152   1467K   175K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 57840.9 total, 57840.9 interval
>Flush(GB): cumulative 0.553, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.49 GB write, 0.11 MB/s write, 6.49 GB read, 0.11 MB/s read, 125.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 07:56:54.020609 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.020603) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:56:54.020614 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 830] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:56:54.020619 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 829 Base level 0, inputs: [10371(2204KB)], [10369(14MB)]
>
>2018-07-11 07:56:54.020632 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814020623, "job": 830, "event": "compaction_started", "files_L0": [10371], "files_L1": [10369], "score": -1, "input_data_size": 17342097}
>2018-07-11 07:56:54.211897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 830] Generated table #10372: 3184 keys, 16406800 bytes
>2018-07-11 07:56:54.211942 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814211915, "cf_name": "default", "job": 830, "event": "table_file_creation", "file_number": 10372, "file_size": 16406800, "table_properties": {"data_size": 16288288, "index_size": 66209, "filter_size": 51376, "raw_key_size": 63998, "raw_average_key_size": 20, "raw_value_size": 16196523, "raw_average_value_size": 5086, "num_data_blocks": 1942, "num_entries": 3184, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:56:54.245099 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 830] Compacted 1@0 + 1@1 files to L1 => 16406800 bytes
>2018-07-11 07:56:54.286946 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.286876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.3 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.4) out(15.6), read-write-amplify(14.9) write-amplify(7.3) OK, records in: 3583, records dropped: 399
>
>2018-07-11 07:56:54.286955 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.286897) EVENT_LOG_v1 {"time_micros": 1531295814286888, "job": 830, "event": "compaction_finished", "compaction_time_micros": 224450, "output_level": 1, "num_output_files": 1, "total_output_size": 16406800, "num_input_records": 3583, "num_output_records": 3184, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:56:54.287476 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814287472, "job": 830, "event": "table_file_deletion", "file_number": 10371}
>2018-07-11 07:56:54.290160 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814290157, "job": 830, "event": "table_file_deletion", "file_number": 10369}
>2018-07-11 07:56:54.290441 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10373. Immutable memtables: 0.
>
>2018-07-11 07:56:54.290480 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 831] Syncing log #10370
>2018-07-11 07:56:54.303931 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.290472) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:56:54.303942 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 831] Flushing memtable with next log file: 10373
>
>2018-07-11 07:56:54.303968 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814303961, "job": 831, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21480}
>2018-07-11 07:56:54.303971 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 831] Level-0 flush table #10374: started
>2018-07-11 07:56:54.347307 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814347294, "cf_name": "default", "job": 831, "event": "table_file_creation", "file_number": 10374, "file_size": 20253, "table_properties": {"data_size": 19183, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18960, "raw_average_value_size": 2370, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:56:54.347329 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 831] Level-0 flush table #10374: 20253 bytes OK
>2018-07-11 07:56:54.397351 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.363858) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10374 started
>2018-07-11 07:56:54.397360 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.397318) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10374: memtable #1 done
>2018-07-11 07:56:54.397362 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.397332) EVENT_LOG_v1 {"time_micros": 1531295814397326, "job": 831, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:56:54.397376 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.397343) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:56:54.397382 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 831] Try to delete WAL files size 19134, prev total WAL file size 19134, number of live WAL files 2.
>
>2018-07-11 07:56:54.397480 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:56:54.397527 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.397520) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 07:56:54.397536 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 832] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:56:54.397541 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 831 Base level 0, inputs: [10374(19KB)], [10372(15MB)]
>
>2018-07-11 07:56:54.397554 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814397545, "job": 832, "event": "compaction_started", "files_L0": [10374], "files_L1": [10372], "score": -1, "input_data_size": 16427053}
>2018-07-11 07:56:54.589777 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 832] Generated table #10375: 3187 keys, 16425931 bytes
>2018-07-11 07:56:54.589827 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814589800, "cf_name": "default", "job": 832, "event": "table_file_creation", "file_number": 10375, "file_size": 16425931, "table_properties": {"data_size": 16307294, "index_size": 66278, "filter_size": 51432, "raw_key_size": 64059, "raw_average_key_size": 20, "raw_value_size": 16215443, "raw_average_value_size": 5087, "num_data_blocks": 1944, "num_entries": 3187, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:56:54.622950 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 832] Compacted 1@0 + 1@1 files to L1 => 16425931 bytes
>2018-07-11 07:56:54.656479 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.656410) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.9 rd, 72.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.6) out(15.7), read-write-amplify(1622.1) write-amplify(811.0) OK, records in: 3192, records dropped: 5
>
>2018-07-11 07:56:54.656488 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:56:54.656432) EVENT_LOG_v1 {"time_micros": 1531295814656422, "job": 832, "event": "compaction_finished", "compaction_time_micros": 225379, "output_level": 1, "num_output_files": 1, "total_output_size": 16425931, "num_input_records": 3192, "num_output_records": 3187, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:56:54.656588 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814656582, "job": 832, "event": "table_file_deletion", "file_number": 10374}
>2018-07-11 07:56:54.659531 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295814659528, "job": 832, "event": "table_file_deletion", "file_number": 10372}
>2018-07-11 07:56:55.007116 7f29149e4700  1 mon.magna102@0(leader).log v20085 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20085)
>2018-07-11 07:56:59.183020 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:56:59.183041 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/162024919' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:56:59.882740 7f29149e4700  1 mon.magna102@0(leader).log v20086 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20086)
>2018-07-11 07:57:00.816097 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:00.816116 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1210965110' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:01.033117 7f29149e4700  1 mon.magna102@0(leader).log v20087 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20087)
>2018-07-11 07:57:02.162678 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:57:02.231876 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:02.231895 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/722863261' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:02.366587 7f29149e4700  1 mon.magna102@0(leader).log v20088 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20088)
>2018-07-11 07:57:03.669199 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:03.669219 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/654919410' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:04.542152 7f29149e4700  1 mon.magna102@0(leader).log v20089 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20089)
>2018-07-11 07:57:05.080393 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:05.080412 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3141543348' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:05.617324 7f29149e4700  1 mon.magna102@0(leader).log v20090 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20090)
>2018-07-11 07:57:06.426690 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:06.426709 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1323370096' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:06.784362 7f29149e4700  1 mon.magna102@0(leader).log v20091 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20091)
>2018-07-11 07:57:07.870318 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:07.870337 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3928243652' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:08.001420 7f29149e4700  1 mon.magna102@0(leader).log v20092 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20092)
>2018-07-11 07:57:09.839158 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:09.839177 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/841697946' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:10.176776 7f29149e4700  1 mon.magna102@0(leader).log v20093 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20093)
>2018-07-11 07:57:11.542481 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:11.542499 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/982197868' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:12.393791 7f29149e4700  1 mon.magna102@0(leader).log v20094 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20094)
>2018-07-11 07:57:15.196602 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 07:57:15.196623 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1822623446' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 07:57:15.902637 7f29149e4700  1 mon.magna102@0(leader).log v20095 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20095)
>2018-07-11 07:57:18.291356 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:18.291374 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/2190651615' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:18.411325 7f29149e4700  1 mon.magna102@0(leader).log v20096 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20096)
>2018-07-11 07:57:20.081502 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:20.081520 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3067735956' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:20.553303 7f29149e4700  1 mon.magna102@0(leader).log v20097 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20097)
>2018-07-11 07:57:51.882084 7f29149e4700  1 mon.magna102@0(leader).log v20098 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20098)
>2018-07-11 07:57:59.655215 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:57:59.655235 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/2716819077' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:57:59.891134 7f29149e4700  1 mon.magna102@0(leader).log v20099 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20099)
>2018-07-11 07:58:02.162851 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 07:58:02.717753 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:58:02.717772 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/547565040' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:58:02.757134 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 07:58:02.891726 7f29149e4700  1 mon.magna102@0(leader).log v20100 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20100)
>2018-07-11 07:58:08.126322 7f29149e4700  1 mon.magna102@0(leader).osd e509 e509: 12 total, 12 up, 12 in
>2018-07-11 07:58:08.167544 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e509: 12 total, 12 up, 12 in
>2018-07-11 07:58:09.259257 7f29149e4700  1 mon.magna102@0(leader).log v20101 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20101)
>2018-07-11 07:58:10.761032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)
>2018-07-11 07:58:10.761077 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 36 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 07:58:10.909631 7f29149e4700  1 mon.magna102@0(leader).log v20102 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20102)
>2018-07-11 07:58:21.902218 7f29149e4700  1 mon.magna102@0(leader).log v20103 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20103)
>2018-07-11 07:58:28.768930 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 07:58:28.929344 7f29149e4700  1 mon.magna102@0(leader).log v20104 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20104)
>2018-07-11 07:58:33.805608 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:58:33.929787 7f29149e4700  1 mon.magna102@0(leader).log v20105 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20105)
>2018-07-11 07:58:42.775072 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:58:42.947342 7f29149e4700  1 mon.magna102@0(leader).log v20106 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20106)
>2018-07-11 07:58:44.947868 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10376. Immutable memtables: 0.
>
>2018-07-11 07:58:44.947913 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 833] Syncing log #10373
>2018-07-11 07:58:44.964441 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:58:44.947902) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 07:58:44.964453 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 833] Flushing memtable with next log file: 10376
>
>2018-07-11 07:58:44.964479 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295924964459, "job": 833, "event": "flush_started", "num_memtables": 1, "num_entries": 834, "num_deletes": 251, "memory_usage": 1393088}
>2018-07-11 07:58:44.964483 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 833] Level-0 flush table #10377: started
>2018-07-11 07:58:45.014968 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925014954, "cf_name": "default", "job": 833, "event": "table_file_creation", "file_number": 10377, "file_size": 1311423, "table_properties": {"data_size": 1300539, "index_size": 5292, "filter_size": 4672, "raw_key_size": 9110, "raw_average_key_size": 20, "raw_value_size": 1290562, "raw_average_value_size": 2900, "num_data_blocks": 161, "num_entries": 445, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 07:58:45.014979 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 833] Level-0 flush table #10377: 1311423 bytes OK
>2018-07-11 07:58:45.073330 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.031504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10377 started
>2018-07-11 07:58:45.073339 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.073295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10377: memtable #1 done
>2018-07-11 07:58:45.073341 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.073309) EVENT_LOG_v1 {"time_micros": 1531295925073303, "job": 833, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 07:58:45.073343 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.073321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 07:58:45.073362 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 833] Try to delete WAL files size 1375087, prev total WAL file size 1375087, number of live WAL files 2.
>
>2018-07-11 07:58:45.073784 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 07:58:45.073848 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.073841) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 07:58:45.073855 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 834] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 07:58:45.073862 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 833 Base level 0, inputs: [10377(1280KB)], [10375(15MB)]
>
>2018-07-11 07:58:45.073874 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925073865, "job": 834, "event": "compaction_started", "files_L0": [10377], "files_L1": [10375], "score": -1, "input_data_size": 17737354}
>2018-07-11 07:58:45.257415 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 834] Generated table #10378: 3118 keys, 15572089 bytes
>2018-07-11 07:58:45.257466 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925257437, "cf_name": "default", "job": 834, "event": "table_file_creation", "file_number": 10378, "file_size": 15572089, "table_properties": {"data_size": 15458658, "index_size": 63495, "filter_size": 49009, "raw_key_size": 62814, "raw_average_key_size": 20, "raw_value_size": 15369667, "raw_average_value_size": 4929, "num_data_blocks": 1854, "num_entries": 3118, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 07:58:45.298929 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 834] Compacted 1@0 + 1@1 files to L1 => 15572089 bytes
>2018-07-11 07:58:45.340775 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.340705) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.8 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.7) out(14.9), read-write-amplify(25.4) write-amplify(11.9) OK, records in: 3632, records dropped: 514
>
>2018-07-11 07:58:45.340784 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-07:58:45.340727) EVENT_LOG_v1 {"time_micros": 1531295925340717, "job": 834, "event": "compaction_finished", "compaction_time_micros": 225037, "output_level": 1, "num_output_files": 1, "total_output_size": 15572089, "num_input_records": 3632, "num_output_records": 3118, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 07:58:45.341144 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925341140, "job": 834, "event": "table_file_deletion", "file_number": 10377}
>2018-07-11 07:58:45.344105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925344102, "job": 834, "event": "table_file_deletion", "file_number": 10375}
>2018-07-11 07:58:48.778166 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:58:48.916643 7f29149e4700  1 mon.magna102@0(leader).log v20107 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20107)
>2018-07-11 07:58:53.813924 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:58:53.908959 7f29149e4700  1 mon.magna102@0(leader).log v20108 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20108)
>2018-07-11 07:59:02.163014 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 07:59:02.784266 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:02.943210 7f29149e4700  1 mon.magna102@0(leader).log v20109 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20109)
>2018-07-11 07:59:12.788782 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:12.945393 7f29149e4700  1 mon.magna102@0(leader).log v20110 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20110)
>2018-07-11 07:59:18.791559 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 40 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:18.945984 7f29149e4700  1 mon.magna102@0(leader).log v20111 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20111)
>2018-07-11 07:59:23.829122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:23.921419 7f29149e4700  1 mon.magna102@0(leader).log v20112 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20112)
>2018-07-11 07:59:31.560651 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:31.560676 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4274910035' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:31.930516 7f29149e4700  1 mon.magna102@0(leader).log v20113 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20113)
>2018-07-11 07:59:32.930528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:33.080884 7f29149e4700  1 mon.magna102@0(leader).log v20114 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20114)
>2018-07-11 07:59:33.976192 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:33.976216 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3935029153' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:34.147717 7f29149e4700  1 mon.magna102@0(leader).log v20115 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20115)
>2018-07-11 07:59:35.516008 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:35.516031 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/519675747' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:36.281408 7f29149e4700  1 mon.magna102@0(leader).log v20116 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20116)
>2018-07-11 07:59:36.975688 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:36.975712 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3262192033' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:37.348242 7f29149e4700  1 mon.magna102@0(leader).log v20117 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20117)
>2018-07-11 07:59:38.801280 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 60 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 07:59:38.957144 7f29149e4700  1 mon.magna102@0(leader).log v20118 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20118)
>2018-07-11 07:59:40.008125 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:40.008148 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2449614357' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:40.132317 7f29149e4700  1 mon.magna102@0(leader).log v20119 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20119)
>2018-07-11 07:59:44.871506 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:44.871531 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4217470096' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:44.872798 7f29191ed700  1 mon.magna102@0(leader).log v20119 check_sub sending message to client.? 10.8.128.102:0/4217470096 with 1 entries (version 20119)
>2018-07-11 07:59:44.991382 7f29149e4700  1 mon.magna102@0(leader).log v20120 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20120)
>2018-07-11 07:59:44.991415 7f29149e4700  1 mon.magna102@0(leader).log v20120 check_sub sending message to client.? 10.8.128.102:0/4217470096 with 0 entries (version 20120)
>2018-07-11 07:59:46.805310 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 59 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 07:59:46.958410 7f29149e4700  1 mon.magna102@0(leader).log v20121 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20121)
>2018-07-11 07:59:46.958456 7f29149e4700  1 mon.magna102@0(leader).log v20121 check_sub sending message to client.? 10.8.128.102:0/4217470096 with 1 entries (version 20121)
>2018-07-11 07:59:48.806128 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 59 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 07:59:48.958728 7f29149e4700  1 mon.magna102@0(leader).log v20122 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20122)
>2018-07-11 07:59:48.958773 7f29149e4700  1 mon.magna102@0(leader).log v20122 check_sub sending message to client.? 10.8.128.102:0/4217470096 with 1 entries (version 20122)
>2018-07-11 07:59:56.809864 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 07:59:56.809914 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering)
>2018-07-11 07:59:56.984476 7f29149e4700  1 mon.magna102@0(leader).log v20123 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20123)
>2018-07-11 07:59:57.819149 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 07:59:57.819175 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1204900659' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 07:59:57.820467 7f29191ed700  1 mon.magna102@0(leader).log v20123 check_sub sending message to client.? 10.8.128.102:0/1204900659 with 1 entries (version 20123)
>2018-07-11 07:59:58.069042 7f29149e4700  1 mon.magna102@0(leader).log v20124 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20124)
>2018-07-11 07:59:58.069075 7f29149e4700  1 mon.magna102@0(leader).log v20124 check_sub sending message to client.? 10.8.128.102:0/1204900659 with 0 entries (version 20124)
>2018-07-11 08:00:02.163178 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:00:05.636235 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:00:05.636262 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/391289955' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:00:05.637519 7f29191ed700  1 mon.magna102@0(leader).log v20124 check_sub sending message to client.? 10.8.128.102:0/391289955 with 1 entries (version 20124)
>2018-07-11 08:00:05.945313 7f29149e4700  1 mon.magna102@0(leader).log v20125 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20125)
>2018-07-11 08:00:05.945346 7f29149e4700  1 mon.magna102@0(leader).log v20125 check_sub sending message to client.? 10.8.128.102:0/391289955 with 0 entries (version 20125)
>2018-07-11 08:00:09.464784 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:00:09.464810 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1194494682' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:00:09.466083 7f29191ed700  1 mon.magna102@0(leader).log v20125 check_sub sending message to client.? 10.8.128.102:0/1194494682 with 1 entries (version 20125)
>2018-07-11 08:00:09.945815 7f29149e4700  1 mon.magna102@0(leader).log v20126 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20126)
>2018-07-11 08:00:09.945847 7f29149e4700  1 mon.magna102@0(leader).log v20126 check_sub sending message to client.? 10.8.128.102:0/1194494682 with 0 entries (version 20126)
>2018-07-11 08:00:12.817566 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:00:12.962990 7f29149e4700  1 mon.magna102@0(leader).log v20127 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20127)
>2018-07-11 08:00:13.879172 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:00:13.879214 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2742891702' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:00:14.071477 7f29149e4700  1 mon.magna102@0(leader).log v20128 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20128)
>2018-07-11 08:00:15.571416 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:00:15.571440 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2124229337' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:00:16.205152 7f29149e4700  1 mon.magna102@0(leader).log v20129 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20129)
>2018-07-11 08:01:02.163371 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:01:11.019345 7f29149e4700  1 mon.magna102@0(leader).log v20130 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20130)
>2018-07-11 08:01:26.851389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 08:01:27.020639 7f29149e4700  1 mon.magna102@0(leader).log v20131 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20131)
>2018-07-11 08:01:28.852247 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 08:01:29.020881 7f29149e4700  1 mon.magna102@0(leader).log v20132 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20132)
>2018-07-11 08:01:52.862556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 08:01:53.014774 7f29149e4700  1 mon.magna102@0(leader).log v20133 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20133)
>2018-07-11 08:01:58.865418 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:01:59.016597 7f29149e4700  1 mon.magna102@0(leader).log v20134 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20134)
>2018-07-11 08:02:00.083406 7f29149e4700  1 mon.magna102@0(leader).log v20135 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20135)
>2018-07-11 08:02:02.163561 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:02:04.192171 7f29149e4700  1 mon.magna102@0(leader).log v20136 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20136)
>2018-07-11 08:02:28.010752 7f29149e4700  1 mon.magna102@0(leader).log v20137 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20137)
>2018-07-11 08:02:48.020607 7f29149e4700  1 mon.magna102@0(leader).log v20138 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20138)
>2018-07-11 08:03:02.163742 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:03:12.898116 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 2,9)
>2018-07-11 08:03:13.048846 7f29149e4700  1 mon.magna102@0(leader).log v20139 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20139)
>2018-07-11 08:03:32.906930 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 08:03:33.058668 7f29149e4700  1 mon.magna102@0(leader).log v20140 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20140)
>2018-07-11 08:03:38.942292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:03:39.059347 7f29149e4700  1 mon.magna102@0(leader).log v20141 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20141)
>2018-07-11 08:04:02.163912 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:04:28.080531 7f29149e4700  1 mon.magna102@0(leader).log v20142 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20142)
>2018-07-11 08:04:52.942414 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 2,9)
>2018-07-11 08:04:53.115973 7f29149e4700  1 mon.magna102@0(leader).log v20143 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20143)
>2018-07-11 08:05:02.164089 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:05:14.051290 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10379. Immutable memtables: 0.
>
>2018-07-11 08:05:14.051336 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 835] Syncing log #10376
>2018-07-11 08:05:14.067507 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.051326) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:05:14.067519 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 835] Flushing memtable with next log file: 10379
>
>2018-07-11 08:05:14.067533 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314067525, "job": 835, "event": "flush_started", "num_memtables": 1, "num_entries": 1909, "num_deletes": 250, "memory_usage": 3750800}
>2018-07-11 08:05:14.067535 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 835] Level-0 flush table #10380: started
>2018-07-11 08:05:14.126511 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314126496, "cf_name": "default", "job": 835, "event": "table_file_creation", "file_number": 10380, "file_size": 2292850, "table_properties": {"data_size": 2275289, "index_size": 8986, "filter_size": 7651, "raw_key_size": 11998, "raw_average_key_size": 21, "raw_value_size": 2260941, "raw_average_value_size": 3966, "num_data_blocks": 278, "num_entries": 570, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:05:14.126521 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 835] Level-0 flush table #10380: 2292850 bytes OK
>2018-07-11 08:05:14.177806 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.143071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10380 started
>2018-07-11 08:05:14.177815 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.177769) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10380: memtable #1 done
>2018-07-11 08:05:14.177817 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.177784) EVENT_LOG_v1 {"time_micros": 1531296314177778, "job": 835, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:05:14.177819 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.177796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:05:14.177825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 835] Try to delete WAL files size 3714237, prev total WAL file size 3714237, number of live WAL files 2.
>
>2018-07-11 08:05:14.178720 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:05:14.178772 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.178765) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:05:14.178781 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 836] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:05:14.178787 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 835 Base level 0, inputs: [10380(2239KB)], [10378(14MB)]
>
>2018-07-11 08:05:14.178800 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314178790, "job": 836, "event": "compaction_started", "files_L0": [10380], "files_L1": [10378], "score": -1, "input_data_size": 17864939}
>2018-07-11 08:05:14.386776 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 836] Generated table #10381: 3368 keys, 17433931 bytes
>2018-07-11 08:05:14.386822 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314386806, "cf_name": "default", "job": 836, "event": "table_file_creation", "file_number": 10381, "file_size": 17433931, "table_properties": {"data_size": 17307810, "index_size": 70561, "filter_size": 54633, "raw_key_size": 67739, "raw_average_key_size": 20, "raw_value_size": 17210388, "raw_average_value_size": 5109, "num_data_blocks": 2069, "num_entries": 3368, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:05:14.420050 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 836] Compacted 1@0 + 1@1 files to L1 => 17433931 bytes
>2018-07-11 08:05:14.453478 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.453408) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.1 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.9) out(16.6), read-write-amplify(15.4) write-amplify(7.6) OK, records in: 3688, records dropped: 320
>
>2018-07-11 08:05:14.453488 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:14.453430) EVENT_LOG_v1 {"time_micros": 1531296314453420, "job": 836, "event": "compaction_finished", "compaction_time_micros": 241236, "output_level": 1, "num_output_files": 1, "total_output_size": 17433931, "num_input_records": 3688, "num_output_records": 3368, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:05:14.453985 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314453981, "job": 836, "event": "table_file_deletion", "file_number": 10380}
>2018-07-11 08:05:14.457023 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296314457021, "job": 836, "event": "table_file_deletion", "file_number": 10378}
>2018-07-11 08:05:14.457078 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:05:16.953252 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 08:05:17.104519 7f29149e4700  1 mon.magna102@0(leader).log v20144 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20144)
>2018-07-11 08:05:22.955746 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:05:23.105119 7f29149e4700  1 mon.magna102@0(leader).log v20145 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20145)
>2018-07-11 08:05:45.107044 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10382. Immutable memtables: 0.
>
>2018-07-11 08:05:45.107084 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 837] Syncing log #10379
>2018-07-11 08:05:45.123654 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.107075) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:05:45.123666 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 837] Flushing memtable with next log file: 10382
>
>2018-07-11 08:05:45.123679 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345123672, "job": 837, "event": "flush_started", "num_memtables": 1, "num_entries": 384, "num_deletes": 251, "memory_usage": 304168}
>2018-07-11 08:05:45.123682 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 837] Level-0 flush table #10383: started
>2018-07-11 08:05:45.165835 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345165822, "cf_name": "default", "job": 837, "event": "table_file_creation", "file_number": 10383, "file_size": 285638, "table_properties": {"data_size": 281959, "index_size": 1232, "filter_size": 1528, "raw_key_size": 6041, "raw_average_key_size": 20, "raw_value_size": 277040, "raw_average_value_size": 929, "num_data_blocks": 38, "num_entries": 298, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:05:45.165845 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 837] Level-0 flush table #10383: 285638 bytes OK
>2018-07-11 08:05:45.215878 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.182427) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10383 started
>2018-07-11 08:05:45.215886 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.215843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10383: memtable #1 done
>2018-07-11 08:05:45.215889 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.215858) EVENT_LOG_v1 {"time_micros": 1531296345215851, "job": 837, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:05:45.215891 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.215869) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:05:45.215896 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 837] Try to delete WAL files size 294483, prev total WAL file size 294483, number of live WAL files 2.
>
>2018-07-11 08:05:45.216035 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:05:45.216084 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.216077) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:05:45.216092 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 838] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:05:45.216098 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 837 Base level 0, inputs: [10383(278KB)], [10381(16MB)]
>
>2018-07-11 08:05:45.216110 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345216101, "job": 838, "event": "compaction_started", "files_L0": [10383], "files_L1": [10381], "score": -1, "input_data_size": 17719569}
>2018-07-11 08:05:45.399937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 838] Generated table #10384: 3156 keys, 15823479 bytes
>2018-07-11 08:05:45.399980 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345399954, "cf_name": "default", "job": 838, "event": "table_file_creation", "file_number": 10384, "file_size": 15823479, "table_properties": {"data_size": 15709407, "index_size": 63559, "filter_size": 49586, "raw_key_size": 63533, "raw_average_key_size": 20, "raw_value_size": 15619841, "raw_average_value_size": 4949, "num_data_blocks": 1855, "num_entries": 3156, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:05:45.433114 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 838] Compacted 1@0 + 1@1 files to L1 => 15823479 bytes
>2018-07-11 08:05:45.466643 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.466577) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.7 rd, 72.9 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.6) out(15.1), read-write-amplify(117.4) write-amplify(55.4) OK, records in: 3666, records dropped: 510
>
>2018-07-11 08:05:45.466652 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:05:45.466598) EVENT_LOG_v1 {"time_micros": 1531296345466589, "job": 838, "event": "compaction_finished", "compaction_time_micros": 216988, "output_level": 1, "num_output_files": 1, "total_output_size": 15823479, "num_input_records": 3666, "num_output_records": 3156, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:05:45.466768 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345466763, "job": 838, "event": "table_file_deletion", "file_number": 10383}
>2018-07-11 08:05:45.469996 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296345469993, "job": 838, "event": "table_file_deletion", "file_number": 10381}
>2018-07-11 08:06:02.164263 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:06:32.987081 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 08:06:33.163583 7f29149e4700  1 mon.magna102@0(leader).log v20146 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20146)
>2018-07-11 08:06:36.989029 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 08:06:37.164032 7f29149e4700  1 mon.magna102@0(leader).log v20147 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20147)
>2018-07-11 08:06:56.997919 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 08:06:57.148894 7f29149e4700  1 mon.magna102@0(leader).log v20148 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20148)
>2018-07-11 08:07:02.164397 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:07:03.000364 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:07:03.150676 7f29149e4700  1 mon.magna102@0(leader).log v20149 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20149)
>2018-07-11 08:07:47.504730 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.504756 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.504782 7f29191ed700  1 mon.magna102@0(leader).osd e509  we're forcing failure of osd.10
>2018-07-11 08:07:47.504816 7f29191ed700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (connection refused reported by osd.6)
>2018-07-11 08:07:47.504971 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.504984 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.505072 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:07:47.505082 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:07:47.505162 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:07:47.505174 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:07:47.505275 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:07:47.505287 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:07:47.505363 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:07:47.505375 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:07:47.505448 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:07:47.505459 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:07:47.505537 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.505548 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.505628 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:07:47.505640 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:07:47.505715 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:07:47.505726 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:07:47.505775 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:07:47.505786 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:07:47.505879 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.505890 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.505962 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:07:47.505974 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:07:47.506040 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:07:47.506051 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:07:47.506121 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:07:47.506133 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:07:47.506210 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:07:47.506222 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:07:47.506299 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:07:47.506310 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:07:47.506381 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:07:47.506392 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:07:47.506463 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.506474 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.506543 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:07:47.506554 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:07:47.506624 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:07:47.506635 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:07:47.506704 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:07:47.506716 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:07:47.506787 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.506798 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.506867 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:07:47.506878 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:07:47.704743 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.704757 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.704813 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.704824 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.704925 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:07:47.704937 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:07:47.705172 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.705202 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.705294 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:07:47.705306 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:07:47.705389 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.705404 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.705481 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:07:47.705492 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:07:47.705570 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:07:47.705582 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:07:47.705666 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:07:47.705677 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:07:47.705749 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:07:47.705760 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:07:47.705830 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:07:47.705842 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:07:47.705911 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:07:47.705925 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:07:47.706005 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:07:47.706017 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:07:47.706088 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:07:47.706099 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:07:47.706169 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:07:47.706186 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:07:47.706260 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:07:47.706272 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:07:47.706345 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:07:47.706357 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:07:47.706429 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:07:47.706441 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:07:47.706515 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:07:47.706526 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:07:47.706596 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:07:47.706607 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:07:47.706679 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:07:47.706690 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:07:47.706762 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:07:47.706773 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:07:47.706842 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:07:47.706853 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:07:47.706925 7f29191ed700  1 mon.magna102@0(leader).osd e509 prepare_failure osd.10 10.8.128.107:6800/14173 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:07:47.706936 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14173 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:07:48.121174 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 08:07:48.154153 7f29149e4700  1 mon.magna102@0(leader).log v20150 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20150)
>2018-07-11 08:07:48.229136 7f29149e4700  1 mon.magna102@0(leader).osd e510 e510: 12 total, 11 up, 12 in
>2018-07-11 08:07:48.263537 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e510: 12 total, 11 up, 12 in
>2018-07-11 08:07:48.404418 7f29149e4700  1 mon.magna102@0(leader).osd e511 e511: 12 total, 11 up, 12 in
>2018-07-11 08:07:48.438310 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e511: 12 total, 11 up, 12 in
>2018-07-11 08:07:49.221375 7f29149e4700  1 mon.magna102@0(leader).log v20151 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20151)
>2018-07-11 08:07:49.404400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:07:49.404446 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 243583/24217500 objects degraded (1.006%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 08:07:50.288411 7f29149e4700  1 mon.magna102@0(leader).log v20152 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20152)
>2018-07-11 08:07:54.172148 7f29149e4700  1 mon.magna102@0(leader).log v20153 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20153)
>2018-07-11 08:07:55.172208 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 841838/24217500 objects degraded (3.476%), 29 pgs degraded (PG_DEGRADED)
>2018-07-11 08:07:55.339233 7f29149e4700  1 mon.magna102@0(leader).log v20154 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20154)
>2018-07-11 08:07:59.064302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 21 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:07:59.181360 7f29149e4700  1 mon.magna102@0(leader).log v20155 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20155)
>2018-07-11 08:08:01.889120 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:01.889144 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/693387199' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:02.164597 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:08:02.191273 7f29149e4700  1 mon.magna102@0(leader).log v20156 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20156)
>2018-07-11 08:08:04.330836 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:04.330860 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3188921512' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:04.466624 7f29149e4700  1 mon.magna102@0(leader).log v20157 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20157)
>2018-07-11 08:08:05.579320 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:05.579344 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2088080415' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:06.633511 7f29149e4700  1 mon.magna102@0(leader).log v20158 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20158)
>2018-07-11 08:08:06.954732 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:06.954755 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1279546261' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:07.817286 7f29149e4700  1 mon.magna102@0(leader).log v20159 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20159)
>2018-07-11 08:08:08.285995 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:08.286018 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3366262303' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:08.892505 7f29149e4700  1 mon.magna102@0(leader).log v20160 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20160)
>2018-07-11 08:08:09.844888 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:09.844911 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3732406109' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:10.059473 7f29149e4700  1 mon.magna102@0(leader).log v20161 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20161)
>2018-07-11 08:08:11.059528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 15 slow requests are blocked > 32 sec. Implicated osds 1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:08:11.234890 7f29149e4700  1 mon.magna102@0(leader).log v20162 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20162)
>2018-07-11 08:08:12.318379 7f29149e4700  1 mon.magna102@0(leader).osd e512 e512: 12 total, 11 up, 12 in
>2018-07-11 08:08:12.360425 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e512: 12 total, 11 up, 12 in
>2018-07-11 08:08:12.472997 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:08:12.473019 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1030793665' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:08:13.318281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 31 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:08:13.485435 7f29149e4700  1 mon.magna102@0(leader).log v20163 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20163)
>2018-07-11 08:08:14.569040 7f29149e4700  1 mon.magna102@0(leader).log v20164 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20164)
>2018-07-11 08:08:15.569035 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1264435/24217500 objects degraded (5.221%), 42 pgs degraded (PG_DEGRADED)
>2018-07-11 08:08:15.744394 7f29149e4700  1 mon.magna102@0(leader).log v20165 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20165)
>2018-07-11 08:08:17.037173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:08:17.194755 7f29149e4700  1 mon.magna102@0(leader).log v20166 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20166)
>2018-07-11 08:08:19.109573 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:08:19.228615 7f29149e4700  1 mon.magna102@0(leader).log v20167 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20167)
>2018-07-11 08:08:21.529041 7f29149e4700  1 mon.magna102@0(leader).osd e513 e513: 12 total, 11 up, 12 in
>2018-07-11 08:08:21.562791 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e513: 12 total, 11 up, 12 in
>2018-07-11 08:08:22.595863 7f29149e4700  1 mon.magna102@0(leader).log v20168 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20168)
>2018-07-11 08:08:22.798745 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 08:08:22.798772 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1071193107' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 08:08:23.595910 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1622009/24217500 objects degraded (6.698%), 50 pgs degraded (PG_DEGRADED)
>2018-07-11 08:08:23.729643 7f29149e4700  1 mon.magna102@0(leader).log v20169 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20169)
>2018-07-11 08:08:24.109806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 5 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:08:24.796517 7f29149e4700  1 mon.magna102@0(leader).log v20170 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20170)
>2018-07-11 08:08:25.796514 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:08:25.796560 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 08:08:25.946883 7f29149e4700  1 mon.magna102@0(leader).log v20171 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20171)
>2018-07-11 08:08:29.110085 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:08:29.110128 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1758027/24217500 objects degraded (7.259%), 52 pgs degraded (PG_DEGRADED)
>2018-07-11 08:08:29.230755 7f29149e4700  1 mon.magna102@0(leader).log v20172 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20172)
>2018-07-11 08:08:31.045213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 13 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:08:31.197725 7f29149e4700  1 mon.magna102@0(leader).log v20173 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20173)
>2018-07-11 08:08:39.110535 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:08:39.249806 7f29149e4700  1 mon.magna102@0(leader).log v20174 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20174)
>2018-07-11 08:08:44.110782 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 37 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:08:44.183578 7f29149e4700  1 mon.magna102@0(leader).log v20175 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20175)
>2018-07-11 08:08:51.056202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:08:51.056250 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1758027/24217500 objects degraded (7.259%), 52 pgs degraded, 31 pgs undersized (PG_DEGRADED)
>2018-07-11 08:08:51.209366 7f29149e4700  1 mon.magna102@0(leader).log v20176 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20176)
>2018-07-11 08:09:02.164786 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:09:13.069298 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs inactive, 11 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:13.069346 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1758027/24217500 objects degraded (7.259%), 52 pgs degraded, 34 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:13.219447 7f29149e4700  1 mon.magna102@0(leader).log v20177 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20177)
>2018-07-11 08:09:19.112111 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1758027/24217500 objects degraded (7.259%), 52 pgs degraded, 53 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:19.228395 7f29149e4700  1 mon.magna102@0(leader).log v20178 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20178)
>2018-07-11 08:09:21.074528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs inactive, 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:21.228723 7f29149e4700  1 mon.magna102@0(leader).log v20179 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20179)
>2018-07-11 08:09:24.112353 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1829002/24217500 objects degraded (7.552%), 54 pgs degraded, 59 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:24.212423 7f29149e4700  1 mon.magna102@0(leader).log v20180 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20180)
>2018-07-11 08:09:25.212444 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 31 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:09:25.362708 7f29149e4700  1 mon.magna102@0(leader).log v20181 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20181)
>2018-07-11 08:09:29.112958 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 19 pgs inactive, 8 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:29.112998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1831696/24217500 objects degraded (7.564%), 55 pgs degraded, 59 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:29.231249 7f29149e4700  1 mon.magna102@0(leader).log v20182 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20182)
>2018-07-11 08:09:32.204554 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 08:09:32.204598 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14489' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 08:09:32.205199 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 08:09:32.205239 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14489' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 08:09:32.205327 7f29191ed700  0 mon.magna102@0(leader).osd e513 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 08:09:32.323464 7f29149e4700  1 mon.magna102@0(leader).log v20183 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20183)
>2018-07-11 08:09:33.324338 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 08:09:33.398734 7f29149e4700  1 mon.magna102@0(leader).osd e514 e514: 12 total, 12 up, 12 in
>2018-07-11 08:09:33.432327 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14489 boot
>2018-07-11 08:09:33.432424 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e514: 12 total, 12 up, 12 in
>2018-07-11 08:09:33.473954 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 08:09:33.473991 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 08:09:33.574265 7f29149e4700  1 mon.magna102@0(leader).log v20184 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20184)
>2018-07-11 08:09:33.649545 7f29149e4700  1 mon.magna102@0(leader).osd e515 e515: 12 total, 12 up, 12 in
>2018-07-11 08:09:33.683647 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e515: 12 total, 12 up, 12 in
>2018-07-11 08:09:34.113243 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 29 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 08:09:34.650220 7f29149e4700  1 mon.magna102@0(leader).log v20185 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20185)
>2018-07-11 08:09:35.650176 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 19 pgs inactive, 11 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:35.650245 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1684994/24217500 objects degraded (6.958%), 52 pgs degraded, 56 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:35.817326 7f29149e4700  1 mon.magna102@0(leader).log v20186 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20186)
>2018-07-11 08:09:44.125967 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 16 pgs inactive, 37 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:44.126012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 512699/24217500 objects degraded (2.117%), 20 pgs degraded, 22 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:44.284890 7f29149e4700  1 mon.magna102@0(leader).log v20187 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20187)
>2018-07-11 08:09:45.284889 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 31 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:09:45.460234 7f29149e4700  1 mon.magna102@0(leader).log v20188 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20188)
>2018-07-11 08:09:45.909713 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:09:45.909736 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/37904156' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:09:46.543728 7f29149e4700  1 mon.magna102@0(leader).log v20189 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20189)
>2018-07-11 08:09:55.097753 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:09:55.269510 7f29149e4700  1 mon.magna102@0(leader).log v20190 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20190)
>2018-07-11 08:09:56.161532 7f29149e4700  1 mon.magna102@0(leader).osd e516 e516: 12 total, 12 up, 12 in
>2018-07-11 08:09:56.203633 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e516: 12 total, 12 up, 12 in
>2018-07-11 08:09:57.161504 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 14 pgs inactive, 40 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:09:57.161551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 321553/24217500 objects degraded (1.328%), 16 pgs degraded, 15 pgs undersized (PG_DEGRADED)
>2018-07-11 08:09:57.328602 7f29149e4700  1 mon.magna102@0(leader).log v20191 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20191)
>2018-07-11 08:09:58.412171 7f29149e4700  1 mon.magna102@0(leader).log v20192 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20192)
>2018-07-11 08:10:02.164952 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:10:03.102425 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 56 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:10:03.281229 7f29149e4700  1 mon.magna102@0(leader).log v20193 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20193)
>2018-07-11 08:10:04.136070 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 14 pgs inactive, 29 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:10:04.356374 7f29149e4700  1 mon.magna102@0(leader).log v20194 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20194)
>2018-07-11 08:10:15.109533 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 73 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:10:15.265841 7f29149e4700  1 mon.magna102@0(leader).log v20195 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20195)
>2018-07-11 08:10:21.112978 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 321553/24217500 objects degraded (1.328%), 16 pgs degraded, 17 pgs undersized (PG_DEGRADED)
>2018-07-11 08:10:21.258108 7f29149e4700  1 mon.magna102@0(leader).log v20196 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20196)
>2018-07-11 08:10:27.116496 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 75 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:10:27.267048 7f29149e4700  1 mon.magna102@0(leader).log v20197 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20197)
>2018-07-11 08:10:29.150367 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 321553/24217500 objects degraded (1.328%), 16 pgs degraded, 18 pgs undersized (PG_DEGRADED)
>2018-07-11 08:10:29.267389 7f29149e4700  1 mon.magna102@0(leader).log v20198 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20198)
>2018-07-11 08:10:35.121341 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 29 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:10:35.276559 7f29149e4700  1 mon.magna102@0(leader).log v20199 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20199)
>2018-07-11 08:10:39.808293 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:39.808316 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/710915930' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:40.260338 7f29149e4700  1 mon.magna102@0(leader).log v20200 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20200)
>2018-07-11 08:10:41.469118 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:41.469142 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1325358370' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:42.393998 7f29149e4700  1 mon.magna102@0(leader).log v20201 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20201)
>2018-07-11 08:10:42.773198 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:42.773222 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3315074577' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:43.527650 7f29149e4700  1 mon.magna102@0(leader).log v20202 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20202)
>2018-07-11 08:10:44.160169 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 32 pgs inactive, 29 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:10:44.594467 7f29149e4700  1 mon.magna102@0(leader).log v20203 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20203)
>2018-07-11 08:10:44.765672 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:44.765695 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/929538695' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:45.728226 7f29149e4700  1 mon.magna102@0(leader).log v20204 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20204)
>2018-07-11 08:10:46.213877 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:46.213901 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1357853330' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:46.795092 7f29149e4700  1 mon.magna102@0(leader).log v20205 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20205)
>2018-07-11 08:10:47.553764 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:47.553787 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2376724851' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:47.928629 7f29149e4700  1 mon.magna102@0(leader).log v20206 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20206)
>2018-07-11 08:10:49.344213 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:10:49.344238 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/696991451' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:10:49.345549 7f29191ed700  1 mon.magna102@0(leader).log v20206 check_sub sending message to client.? 10.8.128.102:0/696991451 with 1 entries (version 20206)
>2018-07-11 08:10:50.288656 7f29149e4700  1 mon.magna102@0(leader).log v20207 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20207)
>2018-07-11 08:10:50.288677 7f29149e4700  1 mon.magna102@0(leader).log v20207 check_sub sending message to client.? 10.8.128.102:0/696991451 with 0 entries (version 20207)
>2018-07-11 08:10:56.706172 7f29149e4700  1 mon.magna102@0(leader).osd e517 e517: 12 total, 12 up, 12 in
>2018-07-11 08:10:56.739842 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e517: 12 total, 12 up, 12 in
>2018-07-11 08:10:57.773054 7f29149e4700  1 mon.magna102@0(leader).log v20208 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20208)
>2018-07-11 08:10:58.848482 7f29149e4700  1 mon.magna102@0(leader).osd e518 e518: 12 total, 12 up, 12 in
>2018-07-11 08:10:58.882219 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e518: 12 total, 12 up, 12 in
>2018-07-11 08:10:59.882027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 31 pgs inactive, 28 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:10:59.915407 7f29149e4700  1 mon.magna102@0(leader).log v20209 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20209)
>2018-07-11 08:11:00.974177 7f29149e4700  1 mon.magna102@0(leader).osd e519 e519: 12 total, 12 up, 12 in
>2018-07-11 08:11:01.007891 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e519: 12 total, 12 up, 12 in
>2018-07-11 08:11:01.074505 7f29149e4700  1 mon.magna102@0(leader).log v20210 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20210)
>2018-07-11 08:11:02.074513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 243893/24217500 objects degraded (1.007%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED)
>2018-07-11 08:11:02.074560 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 68 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:11:02.174663 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:11:02.208203 7f29149e4700  1 mon.magna102@0(leader).log v20211 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20211)
>2018-07-11 08:11:03.275093 7f29149e4700  1 mon.magna102@0(leader).log v20212 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20212)
>2018-07-11 08:11:05.140089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 33 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:11:05.140135 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 243893/24217500 objects degraded (1.007%), 9 pgs degraded, 10 pgs undersized)
>2018-07-11 08:11:05.292218 7f29149e4700  1 mon.magna102@0(leader).log v20213 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20213)
>2018-07-11 08:11:09.175732 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 78 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:11:09.292768 7f29149e4700  1 mon.magna102@0(leader).log v20214 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20214)
>2018-07-11 08:11:14.175938 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 32 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:11:14.276543 7f29149e4700  1 mon.magna102@0(leader).log v20215 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20215)
>2018-07-11 08:11:19.147556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 113 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:11:19.302195 7f29149e4700  1 mon.magna102@0(leader).log v20216 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20216)
>2018-07-11 08:11:23.286407 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10385. Immutable memtables: 0.
>
>2018-07-11 08:11:23.286550 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 839] Syncing log #10382
>2018-07-11 08:11:23.302748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.286540) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:11:23.302759 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 839] Flushing memtable with next log file: 10385
>
>2018-07-11 08:11:23.302774 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683302766, "job": 839, "event": "flush_started", "num_memtables": 1, "num_entries": 2108, "num_deletes": 251, "memory_usage": 5427856}
>2018-07-11 08:11:23.302777 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 839] Level-0 flush table #10386: started
>2018-07-11 08:11:23.386635 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683386607, "cf_name": "default", "job": 839, "event": "table_file_creation", "file_number": 10386, "file_size": 4705980, "table_properties": {"data_size": 4673511, "index_size": 17044, "filter_size": 14499, "raw_key_size": 17417, "raw_average_key_size": 20, "raw_value_size": 4649518, "raw_average_value_size": 5495, "num_data_blocks": 510, "num_entries": 846, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:11:23.386648 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 839] Level-0 flush table #10386: 4705980 bytes OK
>2018-07-11 08:11:23.436642 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.403142) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10386 started
>2018-07-11 08:11:23.436650 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.436600) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10386: memtable #1 done
>2018-07-11 08:11:23.436652 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.436615) EVENT_LOG_v1 {"time_micros": 1531296683436609, "job": 839, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:11:23.436654 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.436631) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:11:23.436661 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 839] Try to delete WAL files size 5383553, prev total WAL file size 5383553, number of live WAL files 2.
>
>2018-07-11 08:11:23.437862 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:11:23.438045 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 08:11:23.438051 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 58710.3 total, 869.4 interval
>Cumulative writes: 81K writes, 385K keys, 81K commit groups, 1.0 writes per commit group, ingest: 0.64 GB, 0.01 MB/s
>Cumulative WAL: 81K writes, 81K syncs, 1.00 writes per sync, written: 0.64 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1175 writes, 5235 keys, 1175 commit groups, 1.0 writes per commit group, ingest: 10.27 MB, 0.01 MB/s
>Interval WAL: 1175 writes, 1175 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    4.49 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        34       420    0.082       0      0
>  L1      1/1   15.09 MB   0.0      6.6     0.6      6.0       6.0     -0.0       0.0  10.8     72.4     66.3        93       418    0.222   1485K   177K
> Sum      2/2   19.58 MB   0.0      6.6     0.6      6.0       6.6      0.6       0.0  11.7     52.9     52.9       127       838    0.152   1485K   177K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.5     55.6     57.6         1        10    0.149     17K   1748
>Uptime(secs): 58710.3 total, 58710.3 interval
>Flush(GB): cumulative 0.561, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.57 GB write, 0.11 MB/s write, 6.57 GB read, 0.11 MB/s read, 127.3 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    4.49 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        34       420    0.082       0      0
>  L1      1/1   15.09 MB   0.0      6.6     0.6      6.0       6.0     -0.0       0.0  10.8     72.4     66.3        93       418    0.222   1485K   177K
> Sum      2/2   19.58 MB   0.0      6.6     0.6      6.0       6.6      0.6       0.0  11.7     52.9     52.9       127       838    0.152   1485K   177K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 58710.3 total, 58710.3 interval
>Flush(GB): cumulative 0.561, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.57 GB write, 0.11 MB/s write, 6.57 GB read, 0.11 MB/s read, 127.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 08:11:23.438072 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.438066) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:11:23.438077 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 840] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:11:23.438082 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 839 Base level 0, inputs: [10386(4595KB)], [10384(15MB)]
>
>2018-07-11 08:11:23.438095 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683438085, "job": 840, "event": "compaction_started", "files_L0": [10386], "files_L1": [10384], "score": -1, "input_data_size": 20529459}
>2018-07-11 08:11:23.655271 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 840] Generated table #10387: 3487 keys, 18092267 bytes
>2018-07-11 08:11:23.655302 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683655288, "cf_name": "default", "job": 840, "event": "table_file_creation", "file_number": 10387, "file_size": 18092267, "table_properties": {"data_size": 17962278, "index_size": 72607, "filter_size": 56455, "raw_key_size": 70572, "raw_average_key_size": 20, "raw_value_size": 17861620, "raw_average_value_size": 5122, "num_data_blocks": 2113, "num_entries": 3487, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:11:23.696776 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 840] Compacted 1@0 + 1@1 files to L1 => 18092267 bytes
>2018-07-11 08:11:23.730283 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.730216) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 79.4 rd, 69.9 wr, level 1, files in(1, 1) out(1) MB in(4.5, 15.1) out(17.3), read-write-amplify(8.2) write-amplify(3.8) OK, records in: 4002, records dropped: 515
>
>2018-07-11 08:11:23.730292 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:11:23.730237) EVENT_LOG_v1 {"time_micros": 1531296683730227, "job": 840, "event": "compaction_finished", "compaction_time_micros": 258664, "output_level": 1, "num_output_files": 1, "total_output_size": 18092267, "num_input_records": 4002, "num_output_records": 3487, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:11:23.731233 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683731229, "job": 840, "event": "table_file_deletion", "file_number": 10386}
>2018-07-11 08:11:23.734280 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296683734277, "job": 840, "event": "table_file_deletion", "file_number": 10384}
>2018-07-11 08:11:24.185384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 121 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:11:24.456624 7f29149e4700  1 mon.magna102@0(leader).log v20217 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20217)
>2018-07-11 08:11:29.189642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 147 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:11:29.323315 7f29149e4700  1 mon.magna102@0(leader).log v20218 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20218)
>2018-07-11 08:11:43.160630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 29 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:11:43.160677 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 146 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,11 (REQUEST_SLOW)
>2018-07-11 08:11:43.316238 7f29149e4700  1 mon.magna102@0(leader).log v20219 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20219)
>2018-07-11 08:11:49.199835 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 145 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:11:49.316816 7f29149e4700  1 mon.magna102@0(leader).log v20220 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20220)
>2018-07-11 08:11:53.433971 7f29149e4700  1 mon.magna102@0(leader).osd e520 e520: 12 total, 12 up, 12 in
>2018-07-11 08:11:53.467689 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e520: 12 total, 12 up, 12 in
>2018-07-11 08:11:54.267478 7f29149e4700  1 mon.magna102@0(leader).osd e521 e521: 12 total, 12 up, 12 in
>2018-07-11 08:11:54.301317 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e521: 12 total, 12 up, 12 in
>2018-07-11 08:11:54.501212 7f29149e4700  1 mon.magna102@0(leader).log v20221 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20221)
>2018-07-11 08:11:55.501260 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 135 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:11:55.651621 7f29149e4700  1 mon.magna102@0(leader).log v20222 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20222)
>2018-07-11 08:11:57.168002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 25 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:11:57.318630 7f29149e4700  1 mon.magna102@0(leader).log v20223 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20223)
>2018-07-11 08:12:01.170187 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 137 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:12:01.335831 7f29149e4700  1 mon.magna102@0(leader).log v20224 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20224)
>2018-07-11 08:12:02.174844 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:12:04.234466 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 26 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:12:04.302758 7f29149e4700  1 mon.magna102@0(leader).log v20225 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20225)
>2018-07-11 08:12:09.234698 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 36 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:12:09.354724 7f29149e4700  1 mon.magna102@0(leader).log v20226 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20226)
>2018-07-11 08:12:12.247515 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:12:12.247538 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3468032500' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:12:12.363354 7f29149e4700  1 mon.magna102@0(leader).log v20227 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20227)
>2018-07-11 08:12:14.352243 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:12:14.352265 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3933746692' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:12:14.513701 7f29149e4700  1 mon.magna102@0(leader).log v20228 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20228)
>2018-07-11 08:12:15.513751 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 156 slow requests are blocked > 32 sec. Implicated osds 0,1,4,11 (REQUEST_SLOW)
>2018-07-11 08:12:15.664042 7f29149e4700  1 mon.magna102@0(leader).log v20229 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20229)
>2018-07-11 08:12:15.717055 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:12:15.717078 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1230713421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:12:16.730937 7f29149e4700  1 mon.magna102@0(leader).log v20230 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20230)
>2018-07-11 08:12:18.172889 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:12:18.172914 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2838642906' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:12:18.864501 7f29149e4700  1 mon.magna102@0(leader).log v20231 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20231)
>2018-07-11 08:12:19.746284 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:12:19.746307 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4205134314' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:12:19.998290 7f29149e4700  1 mon.magna102@0(leader).log v20232 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20232)
>2018-07-11 08:12:24.235299 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 158 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9,11 (REQUEST_SLOW)
>2018-07-11 08:12:24.315449 7f29149e4700  1 mon.magna102@0(leader).log v20233 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20233)
>2018-07-11 08:12:35.190138 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 36 pgs inactive, 22 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:12:35.349936 7f29149e4700  1 mon.magna102@0(leader).log v20234 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20234)
>2018-07-11 08:12:41.193403 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 34 pgs inactive, 21 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:12:41.193449 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 150 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:12:41.350626 7f29149e4700  1 mon.magna102@0(leader).log v20235 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20235)
>2018-07-11 08:12:47.196292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 22 pgs inactive, 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:12:47.351220 7f29149e4700  1 mon.magna102@0(leader).log v20236 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20236)
>2018-07-11 08:12:49.236257 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 57 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 08:12:49.359893 7f29149e4700  1 mon.magna102@0(leader).log v20237 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20237)
>2018-07-11 08:12:59.202526 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:12:59.370352 7f29149e4700  1 mon.magna102@0(leader).log v20238 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20238)
>2018-07-11 08:13:02.175020 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:13:05.205506 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 59 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:13:05.379359 7f29149e4700  1 mon.magna102@0(leader).log v20239 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20239)
>2018-07-11 08:13:29.217482 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 19 pgs inactive, 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:13:29.389480 7f29149e4700  1 mon.magna102@0(leader).log v20240 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20240)
>2018-07-11 08:13:33.219326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 57 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:13:33.364926 7f29149e4700  1 mon.magna102@0(leader).log v20241 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20241)
>2018-07-11 08:13:34.255988 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 17 pgs inactive, 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:13:34.323757 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10388. Immutable memtables: 0.
>
>2018-07-11 08:13:34.323800 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 841] Syncing log #10385
>2018-07-11 08:13:34.340207 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.323791) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:13:34.340218 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 841] Flushing memtable with next log file: 10388
>
>2018-07-11 08:13:34.340244 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814340236, "job": 841, "event": "flush_started", "num_memtables": 1, "num_entries": 930, "num_deletes": 250, "memory_usage": 1954992}
>2018-07-11 08:13:34.340248 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 841] Level-0 flush table #10389: started
>2018-07-11 08:13:34.406134 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814406120, "cf_name": "default", "job": 841, "event": "table_file_creation", "file_number": 10389, "file_size": 1296994, "table_properties": {"data_size": 1287531, "index_size": 4180, "filter_size": 4363, "raw_key_size": 8916, "raw_average_key_size": 21, "raw_value_size": 1278764, "raw_average_value_size": 3081, "num_data_blocks": 129, "num_entries": 415, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:13:34.406144 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 841] Level-0 flush table #10389: 1296994 bytes OK
>2018-07-11 08:13:34.474300 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.440833) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10389 started
>2018-07-11 08:13:34.474309 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.474251) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10389: memtable #1 done
>2018-07-11 08:13:34.474314 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.474265) EVENT_LOG_v1 {"time_micros": 1531296814474259, "job": 841, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:13:34.474316 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.474289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:13:34.474322 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 841] Try to delete WAL files size 1935980, prev total WAL file size 1956170, number of live WAL files 2.
>
>2018-07-11 08:13:34.474491 7f29149e4700  1 mon.magna102@0(leader).log v20242 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20242)
>2018-07-11 08:13:34.474855 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:13:34.474907 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.474900) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:13:34.474917 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 842] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:13:34.474926 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 841 Base level 0, inputs: [10389(1266KB)], [10387(17MB)]
>
>2018-07-11 08:13:34.474939 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814474929, "job": 842, "event": "compaction_started", "files_L0": [10389], "files_L1": [10387], "score": -1, "input_data_size": 19389261}
>2018-07-11 08:13:34.683275 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 842] Generated table #10390: 3449 keys, 18066746 bytes
>2018-07-11 08:13:34.683308 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814683293, "cf_name": "default", "job": 842, "event": "table_file_creation", "file_number": 10390, "file_size": 18066746, "table_properties": {"data_size": 17939750, "index_size": 70181, "filter_size": 55888, "raw_key_size": 69469, "raw_average_key_size": 20, "raw_value_size": 17841420, "raw_average_value_size": 5172, "num_data_blocks": 2048, "num_entries": 3449, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:13:34.724846 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 842] Compacted 1@0 + 1@1 files to L1 => 18066746 bytes
>2018-07-11 08:13:34.766713 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.766642) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 77.6 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(1.2, 17.3) out(17.2), read-write-amplify(28.9) write-amplify(13.9) OK, records in: 3902, records dropped: 453
>
>2018-07-11 08:13:34.766721 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.766664) EVENT_LOG_v1 {"time_micros": 1531296814766654, "job": 842, "event": "compaction_finished", "compaction_time_micros": 249891, "output_level": 1, "num_output_files": 1, "total_output_size": 18066746, "num_input_records": 3902, "num_output_records": 3449, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:13:34.767070 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814767066, "job": 842, "event": "table_file_deletion", "file_number": 10389}
>2018-07-11 08:13:34.770347 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814770332, "job": 842, "event": "table_file_deletion", "file_number": 10387}
>2018-07-11 08:13:34.770605 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10391. Immutable memtables: 0.
>
>2018-07-11 08:13:34.770643 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 843] Syncing log #10388
>2018-07-11 08:13:34.783686 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.770635) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:13:34.783698 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 843] Flushing memtable with next log file: 10391
>
>2018-07-11 08:13:34.783723 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814783716, "job": 843, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 22520}
>2018-07-11 08:13:34.783727 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 843] Level-0 flush table #10392: started
>2018-07-11 08:13:34.825717 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814825704, "cf_name": "default", "job": 843, "event": "table_file_creation", "file_number": 10392, "file_size": 21309, "table_properties": {"data_size": 20239, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 20016, "raw_average_value_size": 2502, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:13:34.825727 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 843] Level-0 flush table #10392: 21309 bytes OK
>2018-07-11 08:13:34.884223 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.842382) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10392 started
>2018-07-11 08:13:34.884232 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.884190) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10392: memtable #1 done
>2018-07-11 08:13:34.884234 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.884204) EVENT_LOG_v1 {"time_micros": 1531296814884198, "job": 843, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:13:34.884248 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.884215) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:13:34.884253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 843] Try to delete WAL files size 20190, prev total WAL file size 20190, number of live WAL files 2.
>
>2018-07-11 08:13:34.884353 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:13:34.884398 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:34.884392) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:13:34.884407 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 844] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:13:34.884413 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 843 Base level 0, inputs: [10392(20KB)], [10390(17MB)]
>
>2018-07-11 08:13:34.884426 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296814884416, "job": 844, "event": "compaction_started", "files_L0": [10392], "files_L1": [10390], "score": -1, "input_data_size": 18088055}
>2018-07-11 08:13:35.093311 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 844] Generated table #10393: 3452 keys, 18086941 bytes
>2018-07-11 08:13:35.093362 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296815093333, "cf_name": "default", "job": 844, "event": "table_file_creation", "file_number": 10393, "file_size": 18086941, "table_properties": {"data_size": 17959815, "index_size": 70250, "filter_size": 55949, "raw_key_size": 69530, "raw_average_key_size": 20, "raw_value_size": 17861396, "raw_average_value_size": 5174, "num_data_blocks": 2050, "num_entries": 3452, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:13:35.136012 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 844] Compacted 1@0 + 1@1 files to L1 => 18086941 bytes
>2018-07-11 08:13:35.169427 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:35.169353) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 71.9 rd, 71.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 17.2) out(17.2), read-write-amplify(1697.6) write-amplify(848.8) OK, records in: 3457, records dropped: 5
>
>2018-07-11 08:13:35.169438 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:13:35.169377) EVENT_LOG_v1 {"time_micros": 1531296815169367, "job": 844, "event": "compaction_finished", "compaction_time_micros": 251570, "output_level": 1, "num_output_files": 1, "total_output_size": 18086941, "num_input_records": 3457, "num_output_records": 3452, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:13:35.169530 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296815169523, "job": 844, "event": "table_file_deletion", "file_number": 10392}
>2018-07-11 08:13:35.173121 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296815173118, "job": 844, "event": "table_file_deletion", "file_number": 10390}
>2018-07-11 08:13:47.226458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 14 pgs inactive, 9 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:13:47.371208 7f29149e4700  1 mon.magna102@0(leader).log v20243 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20243)
>2018-07-11 08:13:51.228353 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 15 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:13:51.371641 7f29149e4700  1 mon.magna102@0(leader).log v20244 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20244)
>2018-07-11 08:13:54.290822 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY)
>2018-07-11 08:13:54.396845 7f29149e4700  1 mon.magna102@0(leader).log v20245 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20245)
>2018-07-11 08:14:02.175309 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:14:05.235060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 39 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:14:05.406450 7f29149e4700  1 mon.magna102@0(leader).log v20246 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20246)
>2018-07-11 08:14:14.291596 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 41 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:14:14.373796 7f29149e4700  1 mon.magna102@0(leader).log v20247 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20247)
>2018-07-11 08:14:21.242467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 40 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:14:21.399536 7f29149e4700  1 mon.magna102@0(leader).log v20248 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20248)
>2018-07-11 08:14:29.292203 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 39 slow requests are blocked > 32 sec. Implicated osds 2,4,9 (REQUEST_SLOW)
>2018-07-11 08:14:29.409996 7f29149e4700  1 mon.magna102@0(leader).log v20249 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20249)
>2018-07-11 08:14:45.253718 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 41 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:14:45.411282 7f29149e4700  1 mon.magna102@0(leader).log v20250 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20250)
>2018-07-11 08:14:47.254530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY)
>2018-07-11 08:14:47.403246 7f29149e4700  1 mon.magna102@0(leader).log v20251 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20251)
>2018-07-11 08:15:02.175484 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:15:23.271289 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 39 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:15:23.422467 7f29149e4700  1 mon.magna102@0(leader).log v20252 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20252)
>2018-07-11 08:15:29.923112 7f29149e4700  1 mon.magna102@0(leader).osd e522 e522: 12 total, 12 up, 12 in
>2018-07-11 08:15:29.956823 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e522: 12 total, 12 up, 12 in
>2018-07-11 08:15:31.006639 7f29149e4700  1 mon.magna102@0(leader).log v20253 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20253)
>2018-07-11 08:15:32.006672 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 6 pgs inactive, 1 pg peering (PG_AVAILABILITY)
>2018-07-11 08:15:32.182049 7f29149e4700  1 mon.magna102@0(leader).log v20254 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20254)
>2018-07-11 08:15:35.276851 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:15:35.425475 7f29149e4700  1 mon.magna102@0(leader).log v20255 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20255)
>2018-07-11 08:15:39.342234 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)
>2018-07-11 08:15:39.509457 7f29149e4700  1 mon.magna102@0(leader).log v20256 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20256)
>2018-07-11 08:15:53.284705 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:15:53.443928 7f29149e4700  1 mon.magna102@0(leader).log v20257 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20257)
>2018-07-11 08:16:01.288307 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,9 (REQUEST_SLOW)
>2018-07-11 08:16:01.436405 7f29149e4700  1 mon.magna102@0(leader).log v20258 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20258)
>2018-07-11 08:16:02.175672 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:16:09.343459 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:16:09.462173 7f29149e4700  1 mon.magna102@0(leader).log v20259 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20259)
>2018-07-11 08:16:25.299049 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 5 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:16:25.446769 7f29149e4700  1 mon.magna102@0(leader).log v20260 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20260)
>2018-07-11 08:17:02.175852 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:17:03.315884 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4 (REQUEST_SLOW)
>2018-07-11 08:17:03.467564 7f29149e4700  1 mon.magna102@0(leader).log v20261 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20261)
>2018-07-11 08:17:13.320392 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:17:13.493443 7f29149e4700  1 mon.magna102@0(leader).log v20262 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20262)
>2018-07-11 08:17:21.324036 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 08:17:21.324082 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:17:21.324100 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering)
>2018-07-11 08:17:21.494205 7f29149e4700  1 mon.magna102@0(leader).log v20263 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20263)
>2018-07-11 08:17:37.330897 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:17:37.478862 7f29149e4700  1 mon.magna102@0(leader).log v20264 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20264)
>2018-07-11 08:17:44.370089 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,9 (REQUEST_SLOW)
>2018-07-11 08:17:44.464017 7f29149e4700  1 mon.magna102@0(leader).log v20265 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20265)
>2018-07-11 08:17:49.372643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:17:49.489644 7f29149e4700  1 mon.magna102@0(leader).log v20266 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20266)
>2018-07-11 08:17:53.473661 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10394. Immutable memtables: 0.
>
>2018-07-11 08:17:53.473706 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 845] Syncing log #10391
>2018-07-11 08:17:53.490234 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.473696) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:17:53.490246 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 845] Flushing memtable with next log file: 10394
>
>2018-07-11 08:17:53.490273 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073490265, "job": 845, "event": "flush_started", "num_memtables": 1, "num_entries": 1375, "num_deletes": 251, "memory_usage": 2698464}
>2018-07-11 08:17:53.490276 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 845] Level-0 flush table #10395: started
>2018-07-11 08:17:53.549107 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073549093, "cf_name": "default", "job": 845, "event": "table_file_creation", "file_number": 10395, "file_size": 2496516, "table_properties": {"data_size": 2476900, "index_size": 10340, "filter_size": 8352, "raw_key_size": 12430, "raw_average_key_size": 20, "raw_value_size": 2461038, "raw_average_value_size": 4081, "num_data_blocks": 313, "num_entries": 603, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:17:53.549118 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 845] Level-0 flush table #10395: 2496516 bytes OK
>2018-07-11 08:17:53.599128 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.565621) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10395 started
>2018-07-11 08:17:53.599136 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.599091) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10395: memtable #1 done
>2018-07-11 08:17:53.599139 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.599105) EVENT_LOG_v1 {"time_micros": 1531297073599099, "job": 845, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:17:53.599154 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.599118) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:17:53.599160 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 845] Try to delete WAL files size 2670569, prev total WAL file size 2670569, number of live WAL files 2.
>
>2018-07-11 08:17:53.599855 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:17:53.599934 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.599926) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:17:53.599946 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 846] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:17:53.599953 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 845 Base level 0, inputs: [10395(2438KB)], [10393(17MB)]
>
>2018-07-11 08:17:53.599968 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073599957, "job": 846, "event": "compaction_started", "files_L0": [10395], "files_L1": [10393], "score": -1, "input_data_size": 20583457}
>2018-07-11 08:17:53.816496 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 846] Generated table #10396: 3539 keys, 18547819 bytes
>2018-07-11 08:17:53.816539 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073816512, "cf_name": "default", "job": 846, "event": "table_file_creation", "file_number": 10396, "file_size": 18547819, "table_properties": {"data_size": 18416837, "index_size": 72614, "filter_size": 57441, "raw_key_size": 71551, "raw_average_key_size": 20, "raw_value_size": 18315464, "raw_average_value_size": 5175, "num_data_blocks": 2112, "num_entries": 3539, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:17:53.849679 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 846] Compacted 1@0 + 1@1 files to L1 => 18547819 bytes
>2018-07-11 08:17:53.883225 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.883150) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 82.4 rd, 74.3 wr, level 1, files in(1, 1) out(1) MB in(2.4, 17.2) out(17.7), read-write-amplify(15.7) write-amplify(7.4) OK, records in: 4055, records dropped: 516
>
>2018-07-11 08:17:53.883234 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:17:53.883174) EVENT_LOG_v1 {"time_micros": 1531297073883162, "job": 846, "event": "compaction_finished", "compaction_time_micros": 249694, "output_level": 1, "num_output_files": 1, "total_output_size": 18547819, "num_input_records": 4055, "num_output_records": 3539, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:17:53.883829 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073883824, "job": 846, "event": "table_file_deletion", "file_number": 10395}
>2018-07-11 08:17:53.887150 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297073887147, "job": 846, "event": "table_file_deletion", "file_number": 10393}
>2018-07-11 08:17:59.340758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,9 (REQUEST_SLOW)
>2018-07-11 08:17:59.492565 7f29149e4700  1 mon.magna102@0(leader).log v20267 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20267)
>2018-07-11 08:18:02.176030 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 08:18:04.375792 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:18:04.476346 7f29149e4700  1 mon.magna102@0(leader).log v20268 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20268)
>2018-07-11 08:18:09.485706 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10397. Immutable memtables: 0.
>
>2018-07-11 08:18:09.485748 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 847] Syncing log #10394
>2018-07-11 08:18:09.502062 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.485739) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:18:09.502073 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 847] Flushing memtable with next log file: 10397
>
>2018-07-11 08:18:09.502099 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089502079, "job": 847, "event": "flush_started", "num_memtables": 1, "num_entries": 577, "num_deletes": 500, "memory_usage": 203728}
>2018-07-11 08:18:09.502102 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 847] Level-0 flush table #10398: started
>2018-07-11 08:18:09.544262 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089544249, "cf_name": "default", "job": 847, "event": "table_file_creation", "file_number": 10398, "file_size": 184409, "table_properties": {"data_size": 180906, "index_size": 762, "filter_size": 1822, "raw_key_size": 10934, "raw_average_key_size": 20, "raw_value_size": 173295, "raw_average_value_size": 326, "num_data_blocks": 24, "num_entries": 531, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 08:18:09.544272 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 847] Level-0 flush table #10398: 184409 bytes OK
>2018-07-11 08:18:09.594303 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.560780) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10398 started
>2018-07-11 08:18:09.594311 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.594268) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10398: memtable #1 done
>2018-07-11 08:18:09.594314 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.594282) EVENT_LOG_v1 {"time_micros": 1531297089594276, "job": 847, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:18:09.594316 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.594294) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:18:09.594335 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 847] Try to delete WAL files size 187936, prev total WAL file size 187936, number of live WAL files 2.
>
>2018-07-11 08:18:09.594464 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:18:09.594515 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.594508) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 08:18:09.594524 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 848] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:18:09.594531 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 847 Base level 0, inputs: [10398(180KB)], [10396(17MB)]
>
>2018-07-11 08:18:09.594543 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089594534, "job": 848, "event": "compaction_started", "files_L0": [10398], "files_L1": [10396], "score": -1, "input_data_size": 18732228}
>2018-07-11 08:18:09.778273 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 848] Generated table #10399: 3062 keys, 15992207 bytes
>2018-07-11 08:18:09.778305 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089778290, "cf_name": "default", "job": 848, "event": "table_file_creation", "file_number": 10399, "file_size": 15992207, "table_properties": {"data_size": 15877599, "index_size": 63902, "filter_size": 49779, "raw_key_size": 61779, "raw_average_key_size": 20, "raw_value_size": 15789186, "raw_average_value_size": 5156, "num_data_blocks": 1868, "num_entries": 3062, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:18:09.811538 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 848] Compacted 1@0 + 1@1 files to L1 => 15992207 bytes
>2018-07-11 08:18:09.845060 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.844993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 86.3 rd, 73.7 wr, level 1, files in(1, 1) out(1) MB in(0.2, 17.7) out(15.3), read-write-amplify(188.3) write-amplify(86.7) OK, records in: 4070, records dropped: 1008
>
>2018-07-11 08:18:09.845069 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:18:09.845015) EVENT_LOG_v1 {"time_micros": 1531297089845006, "job": 848, "event": "compaction_finished", "compaction_time_micros": 216981, "output_level": 1, "num_output_files": 1, "total_output_size": 15992207, "num_input_records": 4070, "num_output_records": 3062, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:18:09.845200 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089845195, "job": 848, "event": "table_file_deletion", "file_number": 10398}
>2018-07-11 08:18:09.848661 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297089848658, "job": 848, "event": "table_file_deletion", "file_number": 10396}
>2018-07-11 08:18:09.848774 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:18:53.364808 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,1,2 (REQUEST_SLOW)
>2018-07-11 08:18:53.517000 7f29149e4700  1 mon.magna102@0(leader).log v20269 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20269)
>2018-07-11 08:18:59.400543 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:18:59.517552 7f29149e4700  1 mon.magna102@0(leader).log v20270 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20270)
>2018-07-11 08:19:02.176216 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:19:11.372799 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:19:11.526912 7f29149e4700  1 mon.magna102@0(leader).log v20271 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20271)
>2018-07-11 08:19:15.374604 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:19:15.528599 7f29149e4700  1 mon.magna102@0(leader).log v20272 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20272)
>2018-07-11 08:19:23.377892 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:19:23.529335 7f29149e4700  1 mon.magna102@0(leader).log v20273 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20273)
>2018-07-11 08:19:35.383300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:19:35.538740 7f29149e4700  1 mon.magna102@0(leader).log v20274 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20274)
>2018-07-11 08:20:02.176401 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:20:37.410390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,1,2 (REQUEST_SLOW)
>2018-07-11 08:20:37.561627 7f29149e4700  1 mon.magna102@0(leader).log v20275 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20275)
>2018-07-11 08:20:43.413130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:20:43.562230 7f29149e4700  1 mon.magna102@0(leader).log v20276 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20276)
>2018-07-11 08:20:49.415943 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,1)
>2018-07-11 08:20:49.562821 7f29149e4700  1 mon.magna102@0(leader).log v20277 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20277)
>2018-07-11 08:21:02.176605 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:21:03.421719 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,9 (REQUEST_SLOW)
>2018-07-11 08:21:03.572359 7f29149e4700  1 mon.magna102@0(leader).log v20278 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20278)
>2018-07-11 08:21:09.424538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:21:09.572971 7f29149e4700  1 mon.magna102@0(leader).log v20279 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20279)
>2018-07-11 08:21:54.569294 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10400. Immutable memtables: 0.
>
>2018-07-11 08:21:54.569337 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 849] Syncing log #10397
>2018-07-11 08:21:54.585786 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.569328) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:21:54.585798 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 849] Flushing memtable with next log file: 10400
>
>2018-07-11 08:21:54.585811 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314585804, "job": 849, "event": "flush_started", "num_memtables": 1, "num_entries": 1134, "num_deletes": 250, "memory_usage": 1971168}
>2018-07-11 08:21:54.585814 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 849] Level-0 flush table #10401: started
>2018-07-11 08:21:54.628010 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314627996, "cf_name": "default", "job": 849, "event": "table_file_creation", "file_number": 10401, "file_size": 1143554, "table_properties": {"data_size": 1133893, "index_size": 4578, "filter_size": 4163, "raw_key_size": 8862, "raw_average_key_size": 21, "raw_value_size": 1124888, "raw_average_value_size": 2717, "num_data_blocks": 143, "num_entries": 414, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:21:54.628021 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 849] Level-0 flush table #10401: 1143554 bytes OK
>2018-07-11 08:21:54.686376 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.644573) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10401 started
>2018-07-11 08:21:54.686385 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.686340) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10401: memtable #1 done
>2018-07-11 08:21:54.686387 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.686355) EVENT_LOG_v1 {"time_micros": 1531297314686349, "job": 849, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:21:54.686389 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.686366) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:21:54.686396 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 849] Try to delete WAL files size 1948297, prev total WAL file size 1948297, number of live WAL files 2.
>
>2018-07-11 08:21:54.686899 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:21:54.687076 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 08:21:54.687083 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 59341.5 total, 631.2 interval
>Cumulative writes: 81K writes, 389K keys, 81K commit groups, 1.0 writes per commit group, ingest: 0.65 GB, 0.01 MB/s
>Cumulative WAL: 81K writes, 81K syncs, 1.00 writes per sync, written: 0.65 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 777 writes, 4024 keys, 777 commit groups, 1.0 writes per commit group, ingest: 6.45 MB, 0.01 MB/s
>Interval WAL: 777 writes, 777 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.09 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        35       425    0.082       0      0
>  L1      1/1   15.25 MB   0.0      6.7     0.6      6.1       6.1     -0.0       0.0  10.8     72.4     66.3        94       423    0.223   1505K   179K
> Sum      2/2   16.34 MB   0.0      6.7     0.6      6.1       6.7      0.6       0.0  11.8     52.9     52.9       129       848    0.152   1505K   179K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  18.3     58.8     56.7         2        10    0.158     19K   2497
>Uptime(secs): 59341.5 total, 59341.5 interval
>Flush(GB): cumulative 0.566, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.66 GB write, 0.11 MB/s write, 6.66 GB read, 0.11 MB/s read, 128.8 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.09 GB read, 0.00 MB/s read, 1.6 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.09 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.7        35       425    0.082       0      0
>  L1      1/1   15.25 MB   0.0      6.7     0.6      6.1       6.1     -0.0       0.0  10.8     72.4     66.3        94       423    0.223   1505K   179K
> Sum      2/2   16.34 MB   0.0      6.7     0.6      6.1       6.7      0.6       0.0  11.8     52.9     52.9       129       848    0.152   1505K   179K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 59341.5 total, 59341.5 interval
>Flush(GB): cumulative 0.566, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.66 GB write, 0.11 MB/s write, 6.66 GB read, 0.11 MB/s read, 128.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 08:21:54.687104 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.687097) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:21:54.687109 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 850] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:21:54.687114 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 849 Base level 0, inputs: [10401(1116KB)], [10399(15MB)]
>
>2018-07-11 08:21:54.687126 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314687117, "job": 850, "event": "compaction_started", "files_L0": [10401], "files_L1": [10399], "score": -1, "input_data_size": 17135761}
>2018-07-11 08:21:54.870399 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 850] Generated table #10402: 3072 keys, 16141635 bytes
>2018-07-11 08:21:54.870431 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314870417, "cf_name": "default", "job": 850, "event": "table_file_creation", "file_number": 10402, "file_size": 16141635, "table_properties": {"data_size": 16027034, "index_size": 63593, "filter_size": 50081, "raw_key_size": 61710, "raw_average_key_size": 20, "raw_value_size": 15938761, "raw_average_value_size": 5188, "num_data_blocks": 1866, "num_entries": 3072, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:21:54.903605 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 850] Compacted 1@0 + 1@1 files to L1 => 16141635 bytes
>2018-07-11 08:21:54.945445 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.945374) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 74.6 wr, level 1, files in(1, 1) out(1) MB in(1.1, 15.3) out(15.4), read-write-amplify(29.1) write-amplify(14.1) OK, records in: 3476, records dropped: 404
>
>2018-07-11 08:21:54.945454 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:21:54.945395) EVENT_LOG_v1 {"time_micros": 1531297314945386, "job": 850, "event": "compaction_finished", "compaction_time_micros": 216463, "output_level": 1, "num_output_files": 1, "total_output_size": 16141635, "num_input_records": 3476, "num_output_records": 3072, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:21:54.945717 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314945713, "job": 850, "event": "table_file_deletion", "file_number": 10401}
>2018-07-11 08:21:54.948730 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297314948728, "job": 850, "event": "table_file_deletion", "file_number": 10399}
>2018-07-11 08:21:54.948784 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:21:59.770010 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:21:59.770038 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:21:59.770069 7f29191ed700  1 mon.magna102@0(leader).osd e522  we're forcing failure of osd.10
>2018-07-11 08:21:59.770091 7f29191ed700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (connection refused reported by osd.7)
>2018-07-11 08:21:59.770300 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:21:59.770312 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:21:59.770369 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:21:59.770381 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:21:59.770487 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:21:59.770498 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:21:59.770579 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:21:59.770590 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:21:59.770668 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:21:59.770679 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:21:59.770751 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:21:59.770762 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:21:59.770835 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:21:59.770847 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:21:59.770923 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:21:59.770934 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:21:59.771006 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:21:59.771017 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:21:59.771088 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:21:59.771100 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:21:59.771171 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:21:59.771189 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:21:59.771264 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:21:59.771275 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:21:59.771345 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:21:59.771356 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:21:59.771427 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:21:59.771438 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:21:59.771509 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:21:59.771520 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:21:59.771590 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:21:59.771601 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:21:59.771669 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:21:59.771680 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:21:59.771749 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:21:59.771760 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:21:59.771830 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:21:59.771841 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:21:59.771912 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:21:59.771923 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:21:59.771994 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:21:59.772005 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:21:59.970425 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:21:59.970444 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:21:59.970554 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:21:59.970566 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:21:59.970628 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:21:59.970643 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:21:59.970751 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:21:59.970763 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:21:59.970840 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:21:59.970851 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:21:59.970931 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:21:59.970941 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:21:59.971012 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:21:59.971023 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:21:59.971091 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:21:59.971103 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:21:59.971173 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:21:59.971195 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:21:59.971274 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:21:59.971286 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:21:59.971357 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:21:59.971368 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:21:59.971453 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:21:59.971464 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:21:59.971538 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:21:59.971550 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:21:59.971625 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:21:59.971636 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:21:59.971707 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:21:59.971718 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:21:59.971790 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:21:59.971801 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:21:59.971873 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:21:59.971884 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:21:59.971954 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:21:59.971965 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:21:59.972035 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:21:59.972046 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:21:59.972115 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:21:59.972126 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:21:59.972208 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:21:59.972219 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:21:59.972291 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:21:59.972303 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:22:00.370293 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:22:00.370308 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:22:00.370364 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 08:22:00.370376 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 08:22:00.371360 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:22:00.371375 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:22:00.371448 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 08:22:00.371460 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 08:22:00.371591 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:22:00.371601 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:22:00.371660 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:22:00.371672 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:22:00.371783 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:22:00.371795 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:22:00.371868 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:22:00.371880 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:22:00.371957 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:22:00.371977 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:22:00.372046 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:22:00.372058 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:22:00.372105 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:22:00.372116 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:22:00.372222 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 08:22:00.372234 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 08:22:00.372316 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 08:22:00.372327 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 08:22:00.372402 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 08:22:00.372413 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 08:22:00.372487 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:22:00.372498 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:22:00.372576 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 08:22:00.372587 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 08:22:00.372658 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 08:22:00.372670 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 08:22:00.372740 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:22:00.372751 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:22:00.372823 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 08:22:00.372835 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 08:22:00.372905 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 08:22:00.372917 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 08:22:00.372986 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 08:22:00.372997 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 08:22:00.373067 7f29191ed700  1 mon.magna102@0(leader).osd e522 prepare_failure osd.10 10.8.128.107:6800/14489 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 08:22:00.373078 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14489 reported immediately failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 08:22:00.547194 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 08:22:00.580295 7f29149e4700  1 mon.magna102@0(leader).log v20280 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20280)
>2018-07-11 08:22:00.655190 7f29149e4700  1 mon.magna102@0(leader).osd e523 e523: 12 total, 11 up, 12 in
>2018-07-11 08:22:00.688726 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e523: 12 total, 11 up, 12 in
>2018-07-11 08:22:01.313726 7f29149e4700  1 mon.magna102@0(leader).osd e524 e524: 12 total, 11 up, 12 in
>2018-07-11 08:22:01.347566 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e524: 12 total, 11 up, 12 in
>2018-07-11 08:22:01.647381 7f29149e4700  1 mon.magna102@0(leader).log v20281 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20281)
>2018-07-11 08:22:02.176742 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:22:03.447977 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 10 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:22:03.448022 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 26121/24217500 objects degraded (0.108%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 08:22:03.606267 7f29149e4700  1 mon.magna102@0(leader).log v20282 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20282)
>2018-07-11 08:22:09.527884 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 23 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:22:09.527929 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 220916/24217500 objects degraded (0.912%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 08:22:10.599752 7f29149e4700  1 mon.magna102@0(leader).log v20283 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20283)
>2018-07-11 08:22:19.350442 7f29149e4700  1 mon.magna102@0(leader).osd e525 e525: 12 total, 11 up, 12 in
>2018-07-11 08:22:19.392519 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e525: 12 total, 11 up, 12 in
>2018-07-11 08:22:20.350423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 0,1,2,9 (REQUEST_SLOW)
>2018-07-11 08:22:20.517548 7f29149e4700  1 mon.magna102@0(leader).log v20284 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20284)
>2018-07-11 08:22:20.934485 7f29149e4700  1 mon.magna102@0(leader).osd e526 e526: 12 total, 11 up, 12 in
>2018-07-11 08:22:20.976534 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e526: 12 total, 11 up, 12 in
>2018-07-11 08:22:21.601450 7f29149e4700  1 mon.magna102@0(leader).log v20285 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20285)
>2018-07-11 08:22:21.934481 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 22 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:22:21.934527 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 289719/24217500 objects degraded (1.196%), 9 pgs degraded (PG_DEGRADED)
>2018-07-11 08:22:22.685138 7f29149e4700  1 mon.magna102@0(leader).log v20286 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20286)
>2018-07-11 08:22:23.852199 7f29149e4700  1 mon.magna102@0(leader).osd e527 e527: 12 total, 11 up, 12 in
>2018-07-11 08:22:23.894252 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e527: 12 total, 11 up, 12 in
>2018-07-11 08:22:24.935744 7f29149e4700  1 mon.magna102@0(leader).log v20287 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20287)
>2018-07-11 08:22:25.969294 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 0,4 (REQUEST_SLOW)
>2018-07-11 08:22:26.011131 7f29149e4700  1 mon.magna102@0(leader).osd e528 e528: 12 total, 11 up, 12 in
>2018-07-11 08:22:26.053176 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e528: 12 total, 11 up, 12 in
>2018-07-11 08:22:26.211622 7f29149e4700  1 mon.magna102@0(leader).log v20288 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20288)
>2018-07-11 08:22:27.278431 7f29149e4700  1 mon.magna102@0(leader).log v20289 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20289)
>2018-07-11 08:22:28.278489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 2 pgs inactive, 13 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:22:28.278536 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1068985/24217500 objects degraded (4.414%), 35 pgs degraded (PG_DEGRADED)
>2018-07-11 08:22:28.428839 7f29149e4700  1 mon.magna102@0(leader).log v20290 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20290)
>2018-07-11 08:22:29.378976 7f29149e4700  1 mon.magna102@0(leader).osd e529 e529: 12 total, 11 up, 12 in
>2018-07-11 08:22:29.412640 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e529: 12 total, 11 up, 12 in
>2018-07-11 08:22:30.529275 7f29149e4700  1 mon.magna102@0(leader).log v20291 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20291)
>2018-07-11 08:22:31.596198 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 08:22:31.746546 7f29149e4700  1 mon.magna102@0(leader).log v20292 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20292)
>2018-07-11 08:22:33.462498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:22:33.462543 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded (PG_DEGRADED)
>2018-07-11 08:22:33.613571 7f29149e4700  1 mon.magna102@0(leader).log v20293 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20293)
>2018-07-11 08:22:34.528922 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 7 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 08:22:34.680373 7f29149e4700  1 mon.magna102@0(leader).log v20294 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20294)
>2018-07-11 08:22:39.529173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 5 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 08:22:39.647762 7f29149e4700  1 mon.magna102@0(leader).log v20295 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20295)
>2018-07-11 08:22:44.529437 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 08:22:44.599491 7f29149e4700  1 mon.magna102@0(leader).log v20296 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20296)
>2018-07-11 08:22:45.908960 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:22:45.908983 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2631102066' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:22:46.733763 7f29149e4700  1 mon.magna102@0(leader).log v20297 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20297)
>2018-07-11 08:22:49.529702 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:22:49.650974 7f29149e4700  1 mon.magna102@0(leader).log v20298 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20298)
>2018-07-11 08:22:54.529992 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:22:54.609759 7f29149e4700  1 mon.magna102@0(leader).log v20299 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20299)
>2018-07-11 08:23:02.176908 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:23:03.479009 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 7 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:03.479054 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded, 5 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:03.635756 7f29149e4700  1 mon.magna102@0(leader).log v20300 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20300)
>2018-07-11 08:23:06.753253 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 08:23:06.753277 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/4158243311' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 08:23:06.877677 7f29149e4700  1 mon.magna102@0(leader).log v20301 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20301)
>2018-07-11 08:23:09.530625 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded, 8 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:09.661585 7f29149e4700  1 mon.magna102@0(leader).log v20302 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20302)
>2018-07-11 08:23:21.489300 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 11 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:21.489346 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded, 20 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:21.646021 7f29149e4700  1 mon.magna102@0(leader).log v20303 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20303)
>2018-07-11 08:23:27.493109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 16 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:27.493154 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded, 43 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:27.646609 7f29149e4700  1 mon.magna102@0(leader).log v20304 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20304)
>2018-07-11 08:23:33.497144 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 31 pgs inactive, 19 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:33.647198 7f29149e4700  1 mon.magna102@0(leader).log v20305 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20305)
>2018-07-11 08:23:34.531613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1585109/24217500 objects degraded (6.545%), 41 pgs degraded, 51 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:34.715270 7f29149e4700  1 mon.magna102@0(leader).log v20306 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20306)
>2018-07-11 08:23:45.158239 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 08:23:45.158272 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14975' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 08:23:45.158927 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 08:23:45.158960 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/14975' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 08:23:45.159060 7f29191ed700  0 mon.magna102@0(leader).osd e529 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 08:23:45.282921 7f29149e4700  1 mon.magna102@0(leader).log v20307 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20307)
>2018-07-11 08:23:46.317322 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 08:23:46.441790 7f29149e4700  1 mon.magna102@0(leader).osd e530 e530: 12 total, 12 up, 12 in
>2018-07-11 08:23:46.483678 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14975 boot
>2018-07-11 08:23:46.483765 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e530: 12 total, 12 up, 12 in
>2018-07-11 08:23:46.525205 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 08:23:46.525231 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 08:23:46.566918 7f29149e4700  1 mon.magna102@0(leader).log v20308 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20308)
>2018-07-11 08:23:46.658917 7f29149e4700  1 mon.magna102@0(leader).osd e531 e531: 12 total, 12 up, 12 in
>2018-07-11 08:23:46.701073 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e531: 12 total, 12 up, 12 in
>2018-07-11 08:23:47.650835 7f29149e4700  1 mon.magna102@0(leader).log v20309 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20309)
>2018-07-11 08:23:49.507803 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 31 pgs inactive, 32 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:49.507848 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1126773/24217500 objects degraded (4.653%), 25 pgs degraded, 34 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:49.676422 7f29149e4700  1 mon.magna102@0(leader).log v20310 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20310)
>2018-07-11 08:23:54.542960 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 48 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:23:54.543006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 266112/24217500 objects degraded (1.099%), 9 pgs degraded, 14 pgs undersized (PG_DEGRADED)
>2018-07-11 08:23:54.676732 7f29149e4700  1 mon.magna102@0(leader).log v20311 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20311)
>2018-07-11 08:23:58.110707 7f29149e4700  1 mon.magna102@0(leader).osd e532 e532: 12 total, 12 up, 12 in
>2018-07-11 08:23:58.152722 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e532: 12 total, 12 up, 12 in
>2018-07-11 08:23:59.194254 7f29149e4700  1 mon.magna102@0(leader).log v20312 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20312)
>2018-07-11 08:24:00.102850 7f29149e4700  1 mon.magna102@0(leader).osd e533 e533: 12 total, 12 up, 12 in
>2018-07-11 08:24:00.144903 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e533: 12 total, 12 up, 12 in
>2018-07-11 08:24:00.194271 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 49 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:24:00.194312 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 191640/24217500 objects degraded (0.791%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED)
>2018-07-11 08:24:01.203337 7f29149e4700  1 mon.magna102@0(leader).log v20313 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20313)
>2018-07-11 08:24:02.177088 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:24:03.703860 7f29149e4700  1 mon.magna102@0(leader).osd e534 e534: 12 total, 12 up, 12 in
>2018-07-11 08:24:03.745854 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e534: 12 total, 12 up, 12 in
>2018-07-11 08:24:04.787422 7f29149e4700  1 mon.magna102@0(leader).log v20314 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20314)
>2018-07-11 08:24:05.787489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 41 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:24:05.787536 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 191146/24217500 objects degraded (0.789%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED)
>2018-07-11 08:24:05.787554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 49 slow requests are blocked > 32 sec. Implicated osds 0,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:24:05.954514 7f29149e4700  1 mon.magna102@0(leader).log v20315 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20315)
>2018-07-11 08:24:06.021560 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10403. Immutable memtables: 0.
>
>2018-07-11 08:24:06.021604 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 851] Syncing log #10400
>2018-07-11 08:24:06.038138 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.021594) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:24:06.038149 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 851] Flushing memtable with next log file: 10403
>
>2018-07-11 08:24:06.038176 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446038168, "job": 851, "event": "flush_started", "num_memtables": 1, "num_entries": 1127, "num_deletes": 251, "memory_usage": 2705200}
>2018-07-11 08:24:06.038187 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 851] Level-0 flush table #10404: started
>2018-07-11 08:24:06.097017 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446097003, "cf_name": "default", "job": 851, "event": "table_file_creation", "file_number": 10404, "file_size": 2302652, "table_properties": {"data_size": 2286621, "index_size": 7797, "filter_size": 7310, "raw_key_size": 11078, "raw_average_key_size": 20, "raw_value_size": 2273490, "raw_average_value_size": 4202, "num_data_blocks": 236, "num_entries": 541, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:24:06.097027 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 851] Level-0 flush table #10404: 2302652 bytes OK
>2018-07-11 08:24:06.148265 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.113527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10404 started
>2018-07-11 08:24:06.148273 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.148229) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10404: memtable #1 done
>2018-07-11 08:24:06.148276 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.148243) EVENT_LOG_v1 {"time_micros": 1531297446148237, "job": 851, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:24:06.148277 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.148255) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:24:06.148296 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 851] Try to delete WAL files size 2681690, prev total WAL file size 2681690, number of live WAL files 2.
>
>2018-07-11 08:24:06.148963 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:24:06.149027 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.149020) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:24:06.149034 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 852] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:24:06.149040 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 851 Base level 0, inputs: [10404(2248KB)], [10402(15MB)]
>
>2018-07-11 08:24:06.149053 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446149043, "job": 852, "event": "compaction_started", "files_L0": [10404], "files_L1": [10402], "score": -1, "input_data_size": 18444287}
>2018-07-11 08:24:06.340636 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 852] Generated table #10405: 3098 keys, 15721515 bytes
>2018-07-11 08:24:06.340685 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446340658, "cf_name": "default", "job": 852, "event": "table_file_creation", "file_number": 10405, "file_size": 15721515, "table_properties": {"data_size": 15608052, "index_size": 63339, "filter_size": 49197, "raw_key_size": 62410, "raw_average_key_size": 20, "raw_value_size": 15519561, "raw_average_value_size": 5009, "num_data_blocks": 1850, "num_entries": 3098, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:24:06.382172 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 852] Compacted 1@0 + 1@1 files to L1 => 15721515 bytes
>2018-07-11 08:24:06.424020 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.423952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(2.2, 15.4) out(15.0), read-write-amplify(14.8) write-amplify(6.8) OK, records in: 3613, records dropped: 515
>
>2018-07-11 08:24:06.424029 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:24:06.423973) EVENT_LOG_v1 {"time_micros": 1531297446423963, "job": 852, "event": "compaction_finished", "compaction_time_micros": 233102, "output_level": 1, "num_output_files": 1, "total_output_size": 15721515, "num_input_records": 3613, "num_output_records": 3098, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:24:06.424589 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446424585, "job": 852, "event": "table_file_deletion", "file_number": 10404}
>2018-07-11 08:24:06.427600 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297446427597, "job": 852, "event": "table_file_deletion", "file_number": 10402}
>2018-07-11 08:24:07.518581 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 191146/24217500 objects degraded (0.789%), 4 pgs degraded, 7 pgs undersized)
>2018-07-11 08:24:07.691442 7f29149e4700  1 mon.magna102@0(leader).log v20316 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20316)
>2018-07-11 08:24:13.521711 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 76 slow requests are blocked > 32 sec. Implicated osds 0,4 (REQUEST_SLOW)
>2018-07-11 08:24:13.675393 7f29149e4700  1 mon.magna102@0(leader).log v20317 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20317)
>2018-07-11 08:24:14.550002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 23 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:24:14.742260 7f29149e4700  1 mon.magna102@0(leader).log v20318 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20318)
>2018-07-11 08:24:19.525072 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 109 slow requests are blocked > 32 sec. Implicated osds 0,4 (REQUEST_SLOW)
>2018-07-11 08:24:19.676389 7f29149e4700  1 mon.magna102@0(leader).log v20319 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20319)
>2018-07-11 08:24:24.559609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 115 slow requests are blocked > 32 sec. Implicated osds 0,4 (REQUEST_SLOW)
>2018-07-11 08:24:24.660252 7f29149e4700  1 mon.magna102@0(leader).log v20320 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20320)
>2018-07-11 08:24:29.560534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 156 slow requests are blocked > 32 sec. Implicated osds 0,4 (REQUEST_SLOW)
>2018-07-11 08:24:29.677581 7f29149e4700  1 mon.magna102@0(leader).log v20321 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20321)
>2018-07-11 08:24:34.560750 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 158 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:24:34.669555 7f29149e4700  1 mon.magna102@0(leader).log v20322 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20322)
>2018-07-11 08:24:43.537746 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 160 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,9 (REQUEST_SLOW)
>2018-07-11 08:24:43.687385 7f29149e4700  1 mon.magna102@0(leader).log v20323 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20323)
>2018-07-11 08:24:47.539954 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 32 pgs inactive, 23 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:24:47.696153 7f29149e4700  1 mon.magna102@0(leader).log v20324 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20324)
>2018-07-11 08:24:54.571337 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 33 pgs inactive, 23 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:24:54.681410 7f29149e4700  1 mon.magna102@0(leader).log v20325 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20325)
>2018-07-11 08:25:02.177225 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:25:09.552092 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 243893/24217500 objects degraded (1.007%), 10 pgs undersized (PG_DEGRADED)
>2018-07-11 08:25:09.707695 7f29149e4700  1 mon.magna102@0(leader).log v20326 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20326)
>2018-07-11 08:25:23.559880 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 159 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4 (REQUEST_SLOW)
>2018-07-11 08:25:23.717172 7f29149e4700  1 mon.magna102@0(leader).log v20327 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20327)
>2018-07-11 08:25:29.600753 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 158 slow requests are blocked > 32 sec. Implicated osds 0,1,4 (REQUEST_SLOW)
>2018-07-11 08:25:29.717658 7f29149e4700  1 mon.magna102@0(leader).log v20328 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20328)
>2018-07-11 08:25:32.776448 7f29149e4700  1 mon.magna102@0(leader).osd e535 e535: 12 total, 12 up, 12 in
>2018-07-11 08:25:32.810130 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e535: 12 total, 12 up, 12 in
>2018-07-11 08:25:33.843391 7f29149e4700  1 mon.magna102@0(leader).osd e536 e536: 12 total, 12 up, 12 in
>2018-07-11 08:25:33.877110 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e536: 12 total, 12 up, 12 in
>2018-07-11 08:25:33.943675 7f29149e4700  1 mon.magna102@0(leader).log v20329 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20329)
>2018-07-11 08:25:35.010503 7f29149e4700  1 mon.magna102@0(leader).log v20330 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20330)
>2018-07-11 08:25:36.043947 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 13 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:25:36.077387 7f29149e4700  1 mon.magna102@0(leader).log v20331 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20331)
>2018-07-11 08:25:37.144454 7f29149e4700  1 mon.magna102@0(leader).log v20332 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20332)
>2018-07-11 08:25:39.403050 7f29149e4700  1 mon.magna102@0(leader).log v20333 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20333)
>2018-07-11 08:25:41.569352 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 18 pgs inactive, 7 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:25:41.569395 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 89 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:25:41.720329 7f29149e4700  1 mon.magna102@0(leader).log v20334 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20334)
>2018-07-11 08:25:45.521649 7f29149e4700  1 mon.magna102@0(leader).osd e537 e537: 12 total, 12 up, 12 in
>2018-07-11 08:25:45.563839 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e537: 12 total, 12 up, 12 in
>2018-07-11 08:25:45.571502 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 08:25:46.605612 7f29149e4700  1 mon.magna102@0(leader).log v20335 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20335)
>2018-07-11 08:25:47.605565 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 16 pgs inactive, 5 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:25:47.780972 7f29149e4700  1 mon.magna102@0(leader).log v20336 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20336)
>2018-07-11 08:25:49.573426 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 243893/24217500 objects degraded (1.007%), 10 pgs undersized)
>2018-07-11 08:25:49.614285 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 88 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:25:49.747999 7f29149e4700  1 mon.magna102@0(leader).log v20337 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20337)
>2018-07-11 08:25:54.614485 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 16 pgs inactive, 12 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:25:54.740103 7f29149e4700  1 mon.magna102@0(leader).log v20338 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20338)
>2018-07-11 08:25:55.740118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:25:55.940464 7f29149e4700  1 mon.magna102@0(leader).log v20339 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20339)
>2018-07-11 08:26:02.177408 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:26:04.615774 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:26:04.716292 7f29149e4700  1 mon.magna102@0(leader).log v20340 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20340)
>2018-07-11 08:26:09.616607 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:26:09.733603 7f29149e4700  1 mon.magna102@0(leader).log v20341 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20341)
>2018-07-11 08:26:14.616816 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 69 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:26:14.717394 7f29149e4700  1 mon.magna102@0(leader).log v20342 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20342)
>2018-07-11 08:26:35.596451 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 73 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:26:35.744201 7f29149e4700  1 mon.magna102@0(leader).log v20343 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20343)
>2018-07-11 08:26:41.599191 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 75 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:26:41.754521 7f29149e4700  1 mon.magna102@0(leader).log v20344 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20344)
>2018-07-11 08:26:49.643707 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 77 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:26:49.763532 7f29149e4700  1 mon.magna102@0(leader).log v20345 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20345)
>2018-07-11 08:26:54.643930 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 83 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:26:54.747335 7f29149e4700  1 mon.magna102@0(leader).log v20346 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20346)
>2018-07-11 08:26:59.647668 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 85 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:26:59.764667 7f29149e4700  1 mon.magna102@0(leader).log v20347 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20347)
>2018-07-11 08:27:02.177584 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:27:04.647922 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 87 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:27:04.748466 7f29149e4700  1 mon.magna102@0(leader).log v20348 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20348)
>2018-07-11 08:27:05.815318 7f29149e4700  1 mon.magna102@0(leader).log v20349 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20349)
>2018-07-11 08:27:09.648961 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 93 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:27:09.765987 7f29149e4700  1 mon.magna102@0(leader).log v20350 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20350)
>2018-07-11 08:27:14.649150 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 97 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:27:14.749769 7f29149e4700  1 mon.magna102@0(leader).log v20351 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20351)
>2018-07-11 08:27:15.910163 7f29149e4700  1 mon.magna102@0(leader).log v20352 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20352)
>2018-07-11 08:27:19.658764 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 102 slow requests are blocked > 32 sec. Implicated osds 0,1,11 (REQUEST_SLOW)
>2018-07-11 08:27:19.775719 7f29149e4700  1 mon.magna102@0(leader).log v20353 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20353)
>2018-07-11 08:27:21.619570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 13 pgs inactive, 10 pgs peering (PG_AVAILABILITY)
>2018-07-11 08:27:21.776042 7f29149e4700  1 mon.magna102@0(leader).log v20354 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20354)
>2018-07-11 08:27:24.658973 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 106 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 08:27:24.759696 7f29149e4700  1 mon.magna102@0(leader).log v20355 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20355)
>2018-07-11 08:27:25.759712 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 10 pgs peering)
>2018-07-11 08:27:25.759757 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 106 slow requests are blocked > 32 sec. Implicated osds 0,11)
>2018-07-11 08:27:25.919638 7f29149e4700  1 mon.magna102@0(leader).log v20356 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20356)
>2018-07-11 08:27:39.627596 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 08:27:39.804100 7f29149e4700  1 mon.magna102@0(leader).log v20357 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20357)
>2018-07-11 08:27:45.630411 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:27:45.796355 7f29149e4700  1 mon.magna102@0(leader).log v20358 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20358)
>2018-07-11 08:27:55.263869 7f29149e4700  1 mon.magna102@0(leader).log v20359 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20359)
>2018-07-11 08:27:59.325357 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 08:27:59.325383 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1159337400' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 08:27:59.326739 7f29191ed700  1 mon.magna102@0(leader).log v20359 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20359)
>2018-07-11 08:27:59.439301 7f29149e4700  1 mon.magna102@0(leader).log v20360 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20360)
>2018-07-11 08:27:59.439321 7f29149e4700  1 mon.magna102@0(leader).log v20360 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20360)
>2018-07-11 08:28:02.177764 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:28:55.661590 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:28:55.819950 7f29149e4700  1 mon.magna102@0(leader).log v20361 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20361)
>2018-07-11 08:28:55.819996 7f29149e4700  1 mon.magna102@0(leader).log v20361 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20361)
>2018-07-11 08:28:59.663415 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:28:59.828738 7f29149e4700  1 mon.magna102@0(leader).log v20362 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20362)
>2018-07-11 08:28:59.828782 7f29149e4700  1 mon.magna102@0(leader).log v20362 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20362)
>2018-07-11 08:29:02.177957 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:29:15.670316 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:29:15.821692 7f29149e4700  1 mon.magna102@0(leader).log v20363 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20363)
>2018-07-11 08:29:15.821739 7f29149e4700  1 mon.magna102@0(leader).log v20363 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20363)
>2018-07-11 08:30:02.178139 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:30:14.794269 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10406. Immutable memtables: 0.
>
>2018-07-11 08:30:14.794312 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 853] Syncing log #10403
>2018-07-11 08:30:14.810670 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.794303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:30:14.810681 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 853] Flushing memtable with next log file: 10406
>
>2018-07-11 08:30:14.810708 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297814810699, "job": 853, "event": "flush_started", "num_memtables": 1, "num_entries": 1953, "num_deletes": 250, "memory_usage": 4383936}
>2018-07-11 08:30:14.810711 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 853] Level-0 flush table #10407: started
>2018-07-11 08:30:14.877894 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297814877880, "cf_name": "default", "job": 853, "event": "table_file_creation", "file_number": 10407, "file_size": 2742165, "table_properties": {"data_size": 2722896, "index_size": 9626, "filter_size": 8719, "raw_key_size": 12740, "raw_average_key_size": 21, "raw_value_size": 2707591, "raw_average_value_size": 4475, "num_data_blocks": 295, "num_entries": 605, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:30:14.877905 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 853] Level-0 flush table #10407: 2742165 bytes OK
>2018-07-11 08:30:14.927909 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.894423) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10407 started
>2018-07-11 08:30:14.927918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.927872) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10407: memtable #1 done
>2018-07-11 08:30:14.927920 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.927887) EVENT_LOG_v1 {"time_micros": 1531297814927880, "job": 853, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:30:14.927922 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.927899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:30:14.927942 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 853] Try to delete WAL files size 4344921, prev total WAL file size 4344921, number of live WAL files 2.
>
>2018-07-11 08:30:14.929022 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:30:14.929084 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:14.929077) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:30:14.929091 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 854] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:30:14.929097 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 853 Base level 0, inputs: [10407(2677KB)], [10405(14MB)]
>
>2018-07-11 08:30:14.929110 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297814929100, "job": 854, "event": "compaction_started", "files_L0": [10407], "files_L1": [10405], "score": -1, "input_data_size": 18463680}
>2018-07-11 08:30:15.136965 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 854] Generated table #10408: 3369 keys, 17956765 bytes
>2018-07-11 08:30:15.137015 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297815136988, "cf_name": "default", "job": 854, "event": "table_file_creation", "file_number": 10408, "file_size": 17956765, "table_properties": {"data_size": 17829540, "index_size": 70653, "filter_size": 55645, "raw_key_size": 67749, "raw_average_key_size": 20, "raw_value_size": 17732138, "raw_average_value_size": 5263, "num_data_blocks": 2072, "num_entries": 3369, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:30:15.170139 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 854] Compacted 1@0 + 1@1 files to L1 => 17956765 bytes
>2018-07-11 08:30:15.203664 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:15.203596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 76.6 rd, 74.5 wr, level 1, files in(1, 1) out(1) MB in(2.6, 15.0) out(17.1), read-write-amplify(13.3) write-amplify(6.5) OK, records in: 3703, records dropped: 334
>
>2018-07-11 08:30:15.203672 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:15.203618) EVENT_LOG_v1 {"time_micros": 1531297815203608, "job": 854, "event": "compaction_finished", "compaction_time_micros": 241010, "output_level": 1, "num_output_files": 1, "total_output_size": 17956765, "num_input_records": 3703, "num_output_records": 3369, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:30:15.204266 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297815204262, "job": 854, "event": "table_file_deletion", "file_number": 10407}
>2018-07-11 08:30:15.207223 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297815207220, "job": 854, "event": "table_file_deletion", "file_number": 10405}
>2018-07-11 08:30:15.207266 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:30:25.701232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:30:25.855202 7f29149e4700  1 mon.magna102@0(leader).log v20364 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20364)
>2018-07-11 08:30:25.855247 7f29149e4700  1 mon.magna102@0(leader).log v20364 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20364)
>2018-07-11 08:30:35.705458 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:30:35.857286 7f29149e4700  1 mon.magna102@0(leader).log v20365 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20365)
>2018-07-11 08:30:35.857332 7f29149e4700  1 mon.magna102@0(leader).log v20365 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20365)
>2018-07-11 08:30:39.849781 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10409. Immutable memtables: 0.
>
>2018-07-11 08:30:39.849823 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 855] Syncing log #10406
>2018-07-11 08:30:39.866279 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.849813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:30:39.866292 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 855] Flushing memtable with next log file: 10409
>
>2018-07-11 08:30:39.866305 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297839866298, "job": 855, "event": "flush_started", "num_memtables": 1, "num_entries": 363, "num_deletes": 251, "memory_usage": 257336}
>2018-07-11 08:30:39.866308 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 855] Level-0 flush table #10410: started
>2018-07-11 08:30:39.900279 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297839900266, "cf_name": "default", "job": 855, "event": "table_file_creation", "file_number": 10410, "file_size": 241542, "table_properties": {"data_size": 238198, "index_size": 1035, "filter_size": 1390, "raw_key_size": 5915, "raw_average_key_size": 20, "raw_value_size": 233507, "raw_average_value_size": 799, "num_data_blocks": 32, "num_entries": 292, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:30:39.900290 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 855] Level-0 flush table #10410: 241542 bytes OK
>2018-07-11 08:30:39.950282 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.916784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10410 started
>2018-07-11 08:30:39.950290 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.950248) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10410: memtable #1 done
>2018-07-11 08:30:39.950293 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.950262) EVENT_LOG_v1 {"time_micros": 1531297839950256, "job": 855, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:30:39.950307 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.950273) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:30:39.950313 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 855] Try to delete WAL files size 248110, prev total WAL file size 248110, number of live WAL files 2.
>
>2018-07-11 08:30:39.950460 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:30:39.950508 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:39.950502) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:30:39.950518 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 856] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:30:39.950524 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 855 Base level 0, inputs: [10410(235KB)], [10408(17MB)]
>
>2018-07-11 08:30:39.950537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297839950528, "job": 856, "event": "compaction_started", "files_L0": [10410], "files_L1": [10408], "score": -1, "input_data_size": 18198307}
>2018-07-11 08:30:40.142673 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 856] Generated table #10411: 3151 keys, 15854749 bytes
>2018-07-11 08:30:40.142722 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297840142694, "cf_name": "default", "job": 856, "event": "table_file_creation", "file_number": 10411, "file_size": 15854749, "table_properties": {"data_size": 15740736, "index_size": 63487, "filter_size": 49599, "raw_key_size": 63417, "raw_average_key_size": 20, "raw_value_size": 15651402, "raw_average_value_size": 4967, "num_data_blocks": 1853, "num_entries": 3151, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:30:40.184152 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 856] Compacted 1@0 + 1@1 files to L1 => 15854749 bytes
>2018-07-11 08:30:40.217603 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:40.217535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.9 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.2, 17.1) out(15.1), read-write-amplify(141.0) write-amplify(65.6) OK, records in: 3661, records dropped: 510
>
>2018-07-11 08:30:40.217612 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:30:40.217558) EVENT_LOG_v1 {"time_micros": 1531297840217548, "job": 856, "event": "compaction_finished", "compaction_time_micros": 233599, "output_level": 1, "num_output_files": 1, "total_output_size": 15854749, "num_input_records": 3661, "num_output_records": 3151, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:30:40.217742 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297840217738, "job": 856, "event": "table_file_deletion", "file_number": 10410}
>2018-07-11 08:30:40.220987 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531297840220984, "job": 856, "event": "table_file_deletion", "file_number": 10408}
>2018-07-11 08:30:49.711466 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:30:49.885725 7f29149e4700  1 mon.magna102@0(leader).log v20366 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20366)
>2018-07-11 08:30:49.885771 7f29149e4700  1 mon.magna102@0(leader).log v20366 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20366)
>2018-07-11 08:31:02.178333 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:32:02.178513 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:32:05.744431 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:32:05.900744 7f29149e4700  1 mon.magna102@0(leader).log v20367 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20367)
>2018-07-11 08:32:05.900789 7f29149e4700  1 mon.magna102@0(leader).log v20367 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20367)
>2018-07-11 08:32:09.746096 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:32:09.901213 7f29149e4700  1 mon.magna102@0(leader).log v20368 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20368)
>2018-07-11 08:32:09.901245 7f29149e4700  1 mon.magna102@0(leader).log v20368 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20368)
>2018-07-11 08:32:25.752950 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:32:25.927454 7f29149e4700  1 mon.magna102@0(leader).log v20369 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20369)
>2018-07-11 08:32:25.927499 7f29149e4700  1 mon.magna102@0(leader).log v20369 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20369)
>2018-07-11 08:33:02.178697 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:33:39.784810 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:33:39.933834 7f29149e4700  1 mon.magna102@0(leader).log v20370 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20370)
>2018-07-11 08:33:39.933879 7f29149e4700  1 mon.magna102@0(leader).log v20370 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20370)
>2018-07-11 08:33:45.787418 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:33:45.943986 7f29149e4700  1 mon.magna102@0(leader).log v20371 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20371)
>2018-07-11 08:33:45.944031 7f29149e4700  1 mon.magna102@0(leader).log v20371 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20371)
>2018-07-11 08:34:02.178872 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:34:05.796030 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:34:05.953840 7f29149e4700  1 mon.magna102@0(leader).log v20372 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20372)
>2018-07-11 08:34:05.953886 7f29149e4700  1 mon.magna102@0(leader).log v20372 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20372)
>2018-07-11 08:35:02.179038 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:35:15.826530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:35:15.976688 7f29149e4700  1 mon.magna102@0(leader).log v20373 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20373)
>2018-07-11 08:35:15.976735 7f29149e4700  1 mon.magna102@0(leader).log v20373 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20373)
>2018-07-11 08:35:19.828306 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:35:19.977289 7f29149e4700  1 mon.magna102@0(leader).log v20374 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20374)
>2018-07-11 08:35:19.977335 7f29149e4700  1 mon.magna102@0(leader).log v20374 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20374)
>2018-07-11 08:35:35.835207 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:35:35.986877 7f29149e4700  1 mon.magna102@0(leader).log v20375 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20375)
>2018-07-11 08:35:35.986923 7f29149e4700  1 mon.magna102@0(leader).log v20375 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20375)
>2018-07-11 08:36:02.179216 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:36:49.867579 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:36:50.018351 7f29149e4700  1 mon.magna102@0(leader).log v20376 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20376)
>2018-07-11 08:36:50.018397 7f29149e4700  1 mon.magna102@0(leader).log v20376 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20376)
>2018-07-11 08:36:55.870032 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:36:56.018932 7f29149e4700  1 mon.magna102@0(leader).log v20377 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20377)
>2018-07-11 08:36:56.018978 7f29149e4700  1 mon.magna102@0(leader).log v20377 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20377)
>2018-07-11 08:37:02.179398 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:37:09.876070 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 08:37:10.028394 7f29149e4700  1 mon.magna102@0(leader).log v20378 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20378)
>2018-07-11 08:37:10.028439 7f29149e4700  1 mon.magna102@0(leader).log v20378 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20378)
>2018-07-11 08:37:14.911522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:37:15.013407 7f29149e4700  1 mon.magna102@0(leader).log v20379 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20379)
>2018-07-11 08:37:15.013453 7f29149e4700  1 mon.magna102@0(leader).log v20379 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20379)
>2018-07-11 08:38:02.179574 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 08:38:25.909400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:38:26.061842 7f29149e4700  1 mon.magna102@0(leader).log v20380 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20380)
>2018-07-11 08:38:26.061890 7f29149e4700  1 mon.magna102@0(leader).log v20380 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20380)
>2018-07-11 08:38:30.045778 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10412. Immutable memtables: 0.
>
>2018-07-11 08:38:30.045823 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 857] Syncing log #10409
>2018-07-11 08:38:30.062368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.045813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:38:30.062379 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 857] Flushing memtable with next log file: 10412
>
>2018-07-11 08:38:30.062406 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310062398, "job": 857, "event": "flush_started", "num_memtables": 1, "num_entries": 2021, "num_deletes": 251, "memory_usage": 3915232}
>2018-07-11 08:38:30.062409 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 857] Level-0 flush table #10413: started
>2018-07-11 08:38:30.137942 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310137915, "cf_name": "default", "job": 857, "event": "table_file_creation", "file_number": 10413, "file_size": 3718799, "table_properties": {"data_size": 3688739, "index_size": 16736, "filter_size": 12399, "raw_key_size": 16032, "raw_average_key_size": 20, "raw_value_size": 3665990, "raw_average_value_size": 4730, "num_data_blocks": 503, "num_entries": 775, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:38:30.137954 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 857] Level-0 flush table #10413: 3718799 bytes OK
>2018-07-11 08:38:30.187930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.154429) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10413 started
>2018-07-11 08:38:30.187938 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.187891) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10413: memtable #1 done
>2018-07-11 08:38:30.187940 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.187906) EVENT_LOG_v1 {"time_micros": 1531298310187899, "job": 857, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:38:30.187956 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.187919) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:38:30.187962 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 857] Try to delete WAL files size 3876366, prev total WAL file size 3876366, number of live WAL files 2.
>
>2018-07-11 08:38:30.188922 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:38:30.189115 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 08:38:30.189122 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 60337.0 total, 995.5 interval
>Cumulative writes: 83K writes, 395K keys, 83K commit groups, 1.0 writes per commit group, ingest: 0.66 GB, 0.01 MB/s
>Cumulative WAL: 83K writes, 83K syncs, 1.00 writes per sync, written: 0.66 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1249 writes, 5464 keys, 1249 commit groups, 1.0 writes per commit group, ingest: 10.63 MB, 0.01 MB/s
>Interval WAL: 1249 writes, 1249 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.55 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.8        35       429    0.082       0      0
>  L1      1/1   15.12 MB   0.0      6.7     0.6      6.2       6.2     -0.0       0.0  10.8     72.5     66.3        95       427    0.223   1519K   181K
> Sum      2/2   18.67 MB   0.0      6.7     0.6      6.2       6.7      0.6       0.0  11.7     53.0     53.0       130       856    0.152   1519K   181K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   8.3     56.2     58.1         1         8    0.153     14K   1763
>Uptime(secs): 60337.0 total, 60337.0 interval
>Flush(GB): cumulative 0.575, interval 0.008
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.73 GB write, 0.11 MB/s write, 6.73 GB read, 0.11 MB/s read, 130.1 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.55 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.8        35       429    0.082       0      0
>  L1      1/1   15.12 MB   0.0      6.7     0.6      6.2       6.2     -0.0       0.0  10.8     72.5     66.3        95       427    0.223   1519K   181K
> Sum      2/2   18.67 MB   0.0      6.7     0.6      6.2       6.7      0.6       0.0  11.7     53.0     53.0       130       856    0.152   1519K   181K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 60337.0 total, 60337.0 interval
>Flush(GB): cumulative 0.575, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.73 GB write, 0.11 MB/s write, 6.73 GB read, 0.11 MB/s read, 130.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 08:38:30.189145 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.189138) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:38:30.189149 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 858] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:38:30.189155 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 857 Base level 0, inputs: [10413(3631KB)], [10411(15MB)]
>
>2018-07-11 08:38:30.189167 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310189158, "job": 858, "event": "compaction_started", "files_L0": [10413], "files_L1": [10411], "score": -1, "input_data_size": 19573548}
>2018-07-11 08:38:30.396952 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 858] Generated table #10414: 3416 keys, 17129635 bytes
>2018-07-11 08:38:30.396995 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310396968, "cf_name": "default", "job": 858, "event": "table_file_creation", "file_number": 10414, "file_size": 17129635, "table_properties": {"data_size": 17002076, "index_size": 72272, "filter_size": 54360, "raw_key_size": 69202, "raw_average_key_size": 20, "raw_value_size": 16902724, "raw_average_value_size": 4948, "num_data_blocks": 2105, "num_entries": 3416, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:38:30.430160 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 858] Compacted 1@0 + 1@1 files to L1 => 17129635 bytes
>2018-07-11 08:38:30.463654 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.463588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.2 rd, 71.1 wr, level 1, files in(1, 1) out(1) MB in(3.5, 15.1) out(16.3), read-write-amplify(9.9) write-amplify(4.6) OK, records in: 3926, records dropped: 510
>
>2018-07-11 08:38:30.463662 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:30.463609) EVENT_LOG_v1 {"time_micros": 1531298310463599, "job": 858, "event": "compaction_finished", "compaction_time_micros": 240975, "output_level": 1, "num_output_files": 1, "total_output_size": 17129635, "num_input_records": 3926, "num_output_records": 3416, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:38:30.464434 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310464429, "job": 858, "event": "table_file_deletion", "file_number": 10413}
>2018-07-11 08:38:30.467413 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298310467397, "job": 858, "event": "table_file_deletion", "file_number": 10411}
>2018-07-11 08:38:33.913086 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:38:34.064540 7f29149e4700  1 mon.magna102@0(leader).log v20381 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20381)
>2018-07-11 08:38:34.064586 7f29149e4700  1 mon.magna102@0(leader).log v20381 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20381)
>2018-07-11 08:38:35.023277 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10415. Immutable memtables: 0.
>
>2018-07-11 08:38:35.023344 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 859] Syncing log #10412
>2018-07-11 08:38:35.039911 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.023334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:38:35.039924 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 859] Flushing memtable with next log file: 10415
>
>2018-07-11 08:38:35.039938 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315039931, "job": 859, "event": "flush_started", "num_memtables": 1, "num_entries": 277, "num_deletes": 250, "memory_usage": 70720}
>2018-07-11 08:38:35.039940 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 859] Level-0 flush table #10416: started
>2018-07-11 08:38:35.082025 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315082012, "cf_name": "default", "job": 859, "event": "table_file_creation", "file_number": 10416, "file_size": 63618, "table_properties": {"data_size": 61632, "index_size": 241, "filter_size": 826, "raw_key_size": 5874, "raw_average_key_size": 22, "raw_value_size": 57853, "raw_average_value_size": 217, "num_data_blocks": 8, "num_entries": 266, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:38:35.082035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 859] Level-0 flush table #10416: 63618 bytes OK
>2018-07-11 08:38:35.132066 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.098574) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10416 started
>2018-07-11 08:38:35.132074 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.132032) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10416: memtable #1 done
>2018-07-11 08:38:35.132077 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.132047) EVENT_LOG_v1 {"time_micros": 1531298315132041, "job": 859, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:38:35.132092 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.132058) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:38:35.132097 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 859] Try to delete WAL files size 63251, prev total WAL file size 63251, number of live WAL files 2.
>
>2018-07-11 08:38:35.132213 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:38:35.132260 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.132254) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:38:35.132269 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 860] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:38:35.132275 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 859 Base level 0, inputs: [10416(62KB)], [10414(16MB)]
>
>2018-07-11 08:38:35.132288 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315132278, "job": 860, "event": "compaction_started", "files_L0": [10416], "files_L1": [10414], "score": -1, "input_data_size": 17193253}
>2018-07-11 08:38:35.316080 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 860] Generated table #10417: 3174 keys, 15474139 bytes
>2018-07-11 08:38:35.316131 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315316102, "cf_name": "default", "job": 860, "event": "table_file_creation", "file_number": 10417, "file_size": 15474139, "table_properties": {"data_size": 15360452, "index_size": 63788, "filter_size": 48972, "raw_key_size": 63867, "raw_average_key_size": 20, "raw_value_size": 15270581, "raw_average_value_size": 4811, "num_data_blocks": 1862, "num_entries": 3174, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:38:35.349217 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 860] Compacted 1@0 + 1@1 files to L1 => 15474139 bytes
>2018-07-11 08:38:35.382816 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.382736) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.3 rd, 71.3 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.3) out(14.8), read-write-amplify(513.5) write-amplify(243.2) OK, records in: 3682, records dropped: 508
>
>2018-07-11 08:38:35.382825 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:38:35.382758) EVENT_LOG_v1 {"time_micros": 1531298315382748, "job": 860, "event": "compaction_finished", "compaction_time_micros": 216914, "output_level": 1, "num_output_files": 1, "total_output_size": 15474139, "num_input_records": 3682, "num_output_records": 3174, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:38:35.382929 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315382925, "job": 860, "event": "table_file_deletion", "file_number": 10416}
>2018-07-11 08:38:35.385962 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298315385959, "job": 860, "event": "table_file_deletion", "file_number": 10414}
>2018-07-11 08:38:35.386080 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:38:43.917585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:38:44.067491 7f29149e4700  1 mon.magna102@0(leader).log v20382 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20382)
>2018-07-11 08:38:44.067536 7f29149e4700  1 mon.magna102@0(leader).log v20382 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20382)
>2018-07-11 08:38:49.920360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)
>2018-07-11 08:38:50.068211 7f29149e4700  1 mon.magna102@0(leader).log v20383 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20383)
>2018-07-11 08:38:50.068245 7f29149e4700  1 mon.magna102@0(leader).log v20383 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20383)
>2018-07-11 08:39:02.179747 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:39:55.949656 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 08:39:56.107428 7f29149e4700  1 mon.magna102@0(leader).log v20384 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20384)
>2018-07-11 08:39:56.107474 7f29149e4700  1 mon.magna102@0(leader).log v20384 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20384)
>2018-07-11 08:40:02.179919 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:40:03.953249 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 08:40:04.117795 7f29149e4700  1 mon.magna102@0(leader).log v20385 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20385)
>2018-07-11 08:40:04.117841 7f29149e4700  1 mon.magna102@0(leader).log v20385 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 1 entries (version 20385)
>2018-07-11 08:40:24.177558 7f29149e4700  1 mon.magna102@0(leader).log v20386 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20386)
>2018-07-11 08:40:24.177591 7f29149e4700  1 mon.magna102@0(leader).log v20386 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20386)
>2018-07-11 08:40:39.103826 7f29149e4700  1 mon.magna102@0(leader).log v20387 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20387)
>2018-07-11 08:40:39.103859 7f29149e4700  1 mon.magna102@0(leader).log v20387 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20387)
>2018-07-11 08:40:49.104654 7f29149e4700  1 mon.magna102@0(leader).log v20388 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20388)
>2018-07-11 08:40:49.104690 7f29149e4700  1 mon.magna102@0(leader).log v20388 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20388)
>2018-07-11 08:41:02.180096 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:41:24.166774 7f29149e4700  1 mon.magna102@0(leader).log v20389 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20389)
>2018-07-11 08:41:24.166808 7f29149e4700  1 mon.magna102@0(leader).log v20389 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20389)
>2018-07-11 08:41:25.233606 7f29149e4700  1 mon.magna102@0(leader).log v20390 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20390)
>2018-07-11 08:41:25.233642 7f29149e4700  1 mon.magna102@0(leader).log v20390 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20390)
>2018-07-11 08:41:27.833873 7f29149e4700  1 mon.magna102@0(leader).log v20391 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20391)
>2018-07-11 08:41:27.833906 7f29149e4700  1 mon.magna102@0(leader).log v20391 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20391)
>2018-07-11 08:41:37.834711 7f29149e4700  1 mon.magna102@0(leader).log v20392 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20392)
>2018-07-11 08:41:37.834743 7f29149e4700  1 mon.magna102@0(leader).log v20392 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20392)
>2018-07-11 08:41:38.968358 7f29149e4700  1 mon.magna102@0(leader).log v20393 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20393)
>2018-07-11 08:41:38.968391 7f29149e4700  1 mon.magna102@0(leader).log v20393 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20393)
>2018-07-11 08:42:02.180268 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:43:02.180410 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:44:02.180591 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:45:02.180776 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:46:02.180923 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:46:22.250635 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10418. Immutable memtables: 0.
>
>2018-07-11 08:46:22.250680 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 861] Syncing log #10415
>2018-07-11 08:46:22.267214 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.250670) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:46:22.267226 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 861] Flushing memtable with next log file: 10418
>
>2018-07-11 08:46:22.267252 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782267244, "job": 861, "event": "flush_started", "num_memtables": 1, "num_entries": 1990, "num_deletes": 251, "memory_usage": 3774272}
>2018-07-11 08:46:22.267255 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 861] Level-0 flush table #10419: started
>2018-07-11 08:46:22.334446 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782334431, "cf_name": "default", "job": 861, "event": "table_file_creation", "file_number": 10419, "file_size": 3605493, "table_properties": {"data_size": 3576037, "index_size": 16438, "filter_size": 12093, "raw_key_size": 15807, "raw_average_key_size": 20, "raw_value_size": 3553654, "raw_average_value_size": 4651, "num_data_blocks": 494, "num_entries": 764, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:46:22.334470 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 861] Level-0 flush table #10419: 3605493 bytes OK
>2018-07-11 08:46:22.384402 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.350931) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10419 started
>2018-07-11 08:46:22.384410 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.384363) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10419: memtable #1 done
>2018-07-11 08:46:22.384413 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.384377) EVENT_LOG_v1 {"time_micros": 1531298782384371, "job": 861, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:46:22.384415 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.384391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:46:22.384434 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 861] Try to delete WAL files size 3735610, prev total WAL file size 3735610, number of live WAL files 2.
>
>2018-07-11 08:46:22.385330 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:46:22.385382 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.385374) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:46:22.385391 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 862] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:46:22.385397 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 861 Base level 0, inputs: [10419(3520KB)], [10417(14MB)]
>
>2018-07-11 08:46:22.385410 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782385400, "job": 862, "event": "compaction_started", "files_L0": [10419], "files_L1": [10417], "score": -1, "input_data_size": 19079632}
>2018-07-11 08:46:22.576780 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 862] Generated table #10420: 3428 keys, 16753369 bytes
>2018-07-11 08:46:22.576830 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782576802, "cf_name": "default", "job": 862, "event": "table_file_creation", "file_number": 10420, "file_size": 16753369, "table_properties": {"data_size": 16626504, "index_size": 72285, "filter_size": 53653, "raw_key_size": 69427, "raw_average_key_size": 20, "raw_value_size": 16526989, "raw_average_value_size": 4821, "num_data_blocks": 2105, "num_entries": 3428, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:46:22.618303 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 862] Compacted 1@0 + 1@1 files to L1 => 16753369 bytes
>2018-07-11 08:46:22.651815 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.651748) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.9 rd, 71.9 wr, level 1, files in(1, 1) out(1) MB in(3.4, 14.8) out(16.0), read-write-amplify(9.9) write-amplify(4.6) OK, records in: 3938, records dropped: 510
>
>2018-07-11 08:46:22.651823 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:22.651769) EVENT_LOG_v1 {"time_micros": 1531298782651760, "job": 862, "event": "compaction_finished", "compaction_time_micros": 232876, "output_level": 1, "num_output_files": 1, "total_output_size": 16753369, "num_input_records": 3938, "num_output_records": 3428, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:46:22.652602 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782652598, "job": 862, "event": "table_file_deletion", "file_number": 10419}
>2018-07-11 08:46:22.655526 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298782655523, "job": 862, "event": "table_file_deletion", "file_number": 10417}
>2018-07-11 08:46:55.229843 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10421. Immutable memtables: 0.
>
>2018-07-11 08:46:55.229884 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 863] Syncing log #10418
>2018-07-11 08:46:55.246405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.229875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:46:55.246416 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 863] Flushing memtable with next log file: 10421
>
>2018-07-11 08:46:55.246442 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815246434, "job": 863, "event": "flush_started", "num_memtables": 1, "num_entries": 367, "num_deletes": 250, "memory_usage": 259928}
>2018-07-11 08:46:55.246445 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 863] Level-0 flush table #10422: started
>2018-07-11 08:46:55.280240 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815280227, "cf_name": "default", "job": 863, "event": "table_file_creation", "file_number": 10422, "file_size": 175908, "table_properties": {"data_size": 173005, "index_size": 794, "filter_size": 1190, "raw_key_size": 6130, "raw_average_key_size": 21, "raw_value_size": 168672, "raw_average_value_size": 604, "num_data_blocks": 25, "num_entries": 279, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:46:55.280249 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 863] Level-0 flush table #10422: 175908 bytes OK
>2018-07-11 08:46:55.338627 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.296789) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10422 started
>2018-07-11 08:46:55.338635 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.338592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10422: memtable #1 done
>2018-07-11 08:46:55.338638 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.338606) EVENT_LOG_v1 {"time_micros": 1531298815338600, "job": 863, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:46:55.338653 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.338617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:46:55.338659 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 863] Try to delete WAL files size 251115, prev total WAL file size 251115, number of live WAL files 2.
>
>2018-07-11 08:46:55.338793 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:46:55.338841 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.338834) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:46:55.338850 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 864] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:46:55.338856 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 863 Base level 0, inputs: [10422(171KB)], [10420(15MB)]
>
>2018-07-11 08:46:55.338871 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815338861, "job": 864, "event": "compaction_started", "files_L0": [10422], "files_L1": [10420], "score": -1, "input_data_size": 16929277}
>2018-07-11 08:46:55.514331 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 864] Generated table #10423: 3211 keys, 15283898 bytes
>2018-07-11 08:46:55.514381 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815514363, "cf_name": "default", "job": 864, "event": "table_file_creation", "file_number": 10423, "file_size": 15283898, "table_properties": {"data_size": 15169440, "index_size": 64679, "filter_size": 48852, "raw_key_size": 64619, "raw_average_key_size": 20, "raw_value_size": 15078422, "raw_average_value_size": 4695, "num_data_blocks": 1888, "num_entries": 3211, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:46:55.548741 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 864] Compacted 1@0 + 1@1 files to L1 => 15283898 bytes
>2018-07-11 08:46:55.582261 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.582195) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.7 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.0) out(14.6), read-write-amplify(183.1) write-amplify(86.9) OK, records in: 3707, records dropped: 496
>
>2018-07-11 08:46:55.582270 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:46:55.582217) EVENT_LOG_v1 {"time_micros": 1531298815582207, "job": 864, "event": "compaction_finished", "compaction_time_micros": 209854, "output_level": 1, "num_output_files": 1, "total_output_size": 15283898, "num_input_records": 3707, "num_output_records": 3211, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:46:55.582386 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815582382, "job": 864, "event": "table_file_deletion", "file_number": 10422}
>2018-07-11 08:46:55.585418 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531298815585415, "job": 864, "event": "table_file_deletion", "file_number": 10420}
>2018-07-11 08:46:55.585536 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:47:02.181093 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:48:02.195393 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:49:02.225696 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:50:02.247938 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:50:35.401794 7f29149e4700  1 mon.magna102@0(leader).log v20394 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20394)
>2018-07-11 08:50:35.401814 7f29149e4700  1 mon.magna102@0(leader).log v20394 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20394)
>2018-07-11 08:50:37.935432 7f29149e4700  1 mon.magna102@0(leader).log v20395 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20395)
>2018-07-11 08:50:37.935464 7f29149e4700  1 mon.magna102@0(leader).log v20395 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20395)
>2018-07-11 08:50:59.403707 7f29149e4700  1 mon.magna102@0(leader).log v20396 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20396)
>2018-07-11 08:50:59.403740 7f29149e4700  1 mon.magna102@0(leader).log v20396 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20396)
>2018-07-11 08:51:02.278898 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:52:02.300776 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:53:02.322704 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:54:02.351558 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:54:34.488826 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10424. Immutable memtables: 0.
>
>2018-07-11 08:54:34.488870 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 865] Syncing log #10421
>2018-07-11 08:54:34.505385 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.488860) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:54:34.505396 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 865] Flushing memtable with next log file: 10424
>
>2018-07-11 08:54:34.505423 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274505415, "job": 865, "event": "flush_started", "num_memtables": 1, "num_entries": 1890, "num_deletes": 251, "memory_usage": 3519664}
>2018-07-11 08:54:34.505426 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 865] Level-0 flush table #10425: started
>2018-07-11 08:54:34.580954 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274580939, "cf_name": "default", "job": 865, "event": "table_file_creation", "file_number": 10425, "file_size": 3361457, "table_properties": {"data_size": 3333631, "index_size": 15541, "filter_size": 11361, "raw_key_size": 15090, "raw_average_key_size": 20, "raw_value_size": 3312366, "raw_average_value_size": 4543, "num_data_blocks": 468, "num_entries": 729, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 08:54:34.580977 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 865] Level-0 flush table #10425: 3361457 bytes OK
>2018-07-11 08:54:34.630947 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.597430) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10425 started
>2018-07-11 08:54:34.630955 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.630909) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10425: memtable #1 done
>2018-07-11 08:54:34.630957 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.630923) EVENT_LOG_v1 {"time_micros": 1531299274630917, "job": 865, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:54:34.630973 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.630937) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:54:34.630979 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 865] Try to delete WAL files size 3482608, prev total WAL file size 3482608, number of live WAL files 2.
>
>2018-07-11 08:54:34.631856 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:54:34.632049 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 08:54:34.632056 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 61301.5 total, 964.4 interval
>Cumulative writes: 84K writes, 399K keys, 84K commit groups, 1.0 writes per commit group, ingest: 0.67 GB, 0.01 MB/s
>Cumulative WAL: 84K writes, 84K syncs, 1.00 writes per sync, written: 0.67 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4524 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.18 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.21 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        35       433    0.081       0      0
>  L1      1/1   14.58 MB   0.0      6.8     0.6      6.2       6.2     -0.0       0.0  10.8     72.5     66.3        96       431    0.223   1534K   183K
> Sum      2/2   17.78 MB   0.0      6.8     0.6      6.2       6.8      0.6       0.0  11.7     53.0     53.0       131       864    0.152   1534K   183K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.0     58.5     57.8         1         8    0.148     15K   2024
>Uptime(secs): 61301.5 total, 61301.5 interval
>Flush(GB): cumulative 0.581, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.80 GB write, 0.11 MB/s write, 6.80 GB read, 0.11 MB/s read, 131.2 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.21 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        35       433    0.081       0      0
>  L1      1/1   14.58 MB   0.0      6.8     0.6      6.2       6.2     -0.0       0.0  10.8     72.5     66.3        96       431    0.223   1534K   183K
> Sum      2/2   17.78 MB   0.0      6.8     0.6      6.2       6.8      0.6       0.0  11.7     53.0     53.0       131       864    0.152   1534K   183K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 61301.5 total, 61301.5 interval
>Flush(GB): cumulative 0.581, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.80 GB write, 0.11 MB/s write, 6.80 GB read, 0.11 MB/s read, 131.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 08:54:34.632077 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.632071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 08:54:34.632083 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 866] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:54:34.632088 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 865 Base level 0, inputs: [10425(3282KB)], [10423(14MB)]
>
>2018-07-11 08:54:34.632101 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274632091, "job": 866, "event": "compaction_started", "files_L0": [10425], "files_L1": [10423], "score": -1, "input_data_size": 18645355}
>2018-07-11 08:54:34.831644 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 866] Generated table #10426: 3430 keys, 16680962 bytes
>2018-07-11 08:54:34.831688 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274831661, "cf_name": "default", "job": 866, "event": "table_file_creation", "file_number": 10426, "file_size": 16680962, "table_properties": {"data_size": 16554275, "index_size": 72252, "filter_size": 53508, "raw_key_size": 69462, "raw_average_key_size": 20, "raw_value_size": 16454760, "raw_average_value_size": 4797, "num_data_blocks": 2104, "num_entries": 3430, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:54:34.873131 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 866] Compacted 1@0 + 1@1 files to L1 => 16680962 bytes
>2018-07-11 08:54:34.906701 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.906633) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.4 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(3.2, 14.6) out(15.9), read-write-amplify(10.5) write-amplify(5.0) OK, records in: 3940, records dropped: 510
>
>2018-07-11 08:54:34.906710 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:54:34.906654) EVENT_LOG_v1 {"time_micros": 1531299274906645, "job": 866, "event": "compaction_finished", "compaction_time_micros": 241016, "output_level": 1, "num_output_files": 1, "total_output_size": 16680962, "num_input_records": 3940, "num_output_records": 3430, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:54:34.907425 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274907420, "job": 866, "event": "table_file_deletion", "file_number": 10425}
>2018-07-11 08:54:34.910334 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299274910331, "job": 866, "event": "table_file_deletion", "file_number": 10423}
>2018-07-11 08:55:02.375472 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 08:55:15.444702 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10427. Immutable memtables: 0.
>
>2018-07-11 08:55:15.444743 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 867] Syncing log #10424
>2018-07-11 08:55:15.461288 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.444734) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 08:55:15.461299 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 867] Flushing memtable with next log file: 10427
>
>2018-07-11 08:55:15.461325 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315461317, "job": 867, "event": "flush_started", "num_memtables": 1, "num_entries": 395, "num_deletes": 250, "memory_usage": 319880}
>2018-07-11 08:55:15.461328 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 867] Level-0 flush table #10428: started
>2018-07-11 08:55:15.495141 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315495128, "cf_name": "default", "job": 867, "event": "table_file_creation", "file_number": 10428, "file_size": 205809, "table_properties": {"data_size": 202686, "index_size": 922, "filter_size": 1282, "raw_key_size": 6210, "raw_average_key_size": 21, "raw_value_size": 198204, "raw_average_value_size": 700, "num_data_blocks": 29, "num_entries": 283, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 08:55:15.495151 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 867] Level-0 flush table #10428: 205809 bytes OK
>2018-07-11 08:55:15.545176 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.511659) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10428 started
>2018-07-11 08:55:15.545191 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.545141) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10428: memtable #1 done
>2018-07-11 08:55:15.545206 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.545155) EVENT_LOG_v1 {"time_micros": 1531299315545149, "job": 867, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 08:55:15.545208 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.545167) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 08:55:15.545215 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 867] Try to delete WAL files size 310447, prev total WAL file size 310447, number of live WAL files 2.
>
>2018-07-11 08:55:15.545362 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:55:15.545409 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.545402) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 08:55:15.545418 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 868] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 08:55:15.545424 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 867 Base level 0, inputs: [10428(200KB)], [10426(15MB)]
>
>2018-07-11 08:55:15.545437 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315545427, "job": 868, "event": "compaction_started", "files_L0": [10428], "files_L1": [10426], "score": -1, "input_data_size": 16886771}
>2018-07-11 08:55:15.729257 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 868] Generated table #10429: 3221 keys, 15270514 bytes
>2018-07-11 08:55:15.729302 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315729274, "cf_name": "default", "job": 868, "event": "table_file_creation", "file_number": 10429, "file_size": 15270514, "table_properties": {"data_size": 15155771, "index_size": 64917, "filter_size": 48899, "raw_key_size": 64822, "raw_average_key_size": 20, "raw_value_size": 15064449, "raw_average_value_size": 4676, "num_data_blocks": 1895, "num_entries": 3221, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 08:55:15.770768 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 868] Compacted 1@0 + 1@1 files to L1 => 15270514 bytes
>2018-07-11 08:55:15.804254 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.804185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.9 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.9) out(14.6), read-write-amplify(156.2) write-amplify(74.2) OK, records in: 3713, records dropped: 492
>
>2018-07-11 08:55:15.804263 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-08:55:15.804208) EVENT_LOG_v1 {"time_micros": 1531299315804199, "job": 868, "event": "compaction_finished", "compaction_time_micros": 225314, "output_level": 1, "num_output_files": 1, "total_output_size": 15270514, "num_input_records": 3713, "num_output_records": 3221, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 08:55:15.804386 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315804381, "job": 868, "event": "table_file_deletion", "file_number": 10428}
>2018-07-11 08:55:15.807355 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299315807352, "job": 868, "event": "table_file_deletion", "file_number": 10426}
>2018-07-11 08:55:15.807468 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 08:56:02.407628 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:57:02.437942 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:58:02.459736 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:59:02.481674 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 08:59:37.909140 7f29149e4700  1 mon.magna102@0(leader).log v20397 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20397)
>2018-07-11 08:59:37.909160 7f29149e4700  1 mon.magna102@0(leader).log v20397 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20397)
>2018-07-11 08:59:38.975968 7f29149e4700  1 mon.magna102@0(leader).log v20398 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20398)
>2018-07-11 08:59:38.975989 7f29149e4700  1 mon.magna102@0(leader).log v20398 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20398)
>2018-07-11 08:59:58.237055 7f29149e4700  1 mon.magna102@0(leader).log v20399 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20399)
>2018-07-11 08:59:58.237075 7f29149e4700  1 mon.magna102@0(leader).log v20399 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20399)
>2018-07-11 08:59:59.303884 7f29149e4700  1 mon.magna102@0(leader).log v20400 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20400)
>2018-07-11 08:59:59.303905 7f29149e4700  1 mon.magna102@0(leader).log v20400 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20400)
>2018-07-11 09:00:02.521185 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:00:03.763210 7f29149e4700  1 mon.magna102@0(leader).log v20401 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20401)
>2018-07-11 09:00:03.763243 7f29149e4700  1 mon.magna102@0(leader).log v20401 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20401)
>2018-07-11 09:00:13.764120 7f29149e4700  1 mon.magna102@0(leader).log v20402 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20402)
>2018-07-11 09:00:13.764152 7f29149e4700  1 mon.magna102@0(leader).log v20402 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20402)
>2018-07-11 09:00:17.631244 7f29149e4700  1 mon.magna102@0(leader).log v20403 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20403)
>2018-07-11 09:00:17.631278 7f29149e4700  1 mon.magna102@0(leader).log v20403 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20403)
>2018-07-11 09:00:27.648734 7f29149e4700  1 mon.magna102@0(leader).log v20404 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20404)
>2018-07-11 09:00:27.648767 7f29149e4700  1 mon.magna102@0(leader).log v20404 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20404)
>2018-07-11 09:00:29.815702 7f29149e4700  1 mon.magna102@0(leader).log v20405 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20405)
>2018-07-11 09:00:29.815737 7f29149e4700  1 mon.magna102@0(leader).log v20405 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20405)
>2018-07-11 09:00:39.658419 7f29149e4700  1 mon.magna102@0(leader).log v20406 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20406)
>2018-07-11 09:00:39.658440 7f29149e4700  1 mon.magna102@0(leader).log v20406 check_sub sending message to client.? 10.8.128.102:0/1159337400 with 0 entries (version 20406)
>2018-07-11 09:01:02.544794 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:02:02.565410 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:02:32.710805 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10430. Immutable memtables: 0.
>
>2018-07-11 09:02:32.710850 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 869] Syncing log #10427
>2018-07-11 09:02:32.727376 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.710840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:02:32.727387 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 869] Flushing memtable with next log file: 10430
>
>2018-07-11 09:02:32.727414 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299752727406, "job": 869, "event": "flush_started", "num_memtables": 1, "num_entries": 1869, "num_deletes": 251, "memory_usage": 3465312}
>2018-07-11 09:02:32.727418 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 869] Level-0 flush table #10431: started
>2018-07-11 09:02:32.794582 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299752794567, "cf_name": "default", "job": 869, "event": "table_file_creation", "file_number": 10431, "file_size": 3313455, "table_properties": {"data_size": 3286041, "index_size": 15276, "filter_size": 11214, "raw_key_size": 15055, "raw_average_key_size": 20, "raw_value_size": 3264981, "raw_average_value_size": 4484, "num_data_blocks": 460, "num_entries": 728, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:02:32.794606 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 869] Level-0 flush table #10431: 3313455 bytes OK
>2018-07-11 09:02:32.844592 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.811098) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10431 started
>2018-07-11 09:02:32.844600 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.844554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10431: memtable #1 done
>2018-07-11 09:02:32.844603 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.844569) EVENT_LOG_v1 {"time_micros": 1531299752844563, "job": 869, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:02:32.844605 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.844581) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:02:32.844624 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 869] Try to delete WAL files size 3428752, prev total WAL file size 3428752, number of live WAL files 2.
>
>2018-07-11 09:02:32.845559 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:02:32.845622 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:02:32.845615) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:02:32.845629 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 870] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:02:32.845635 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 869 Base level 0, inputs: [10431(3235KB)], [10429(14MB)]
>
>2018-07-11 09:02:32.845648 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299752845639, "job": 870, "event": "compaction_started", "files_L0": [10431], "files_L1": [10429], "score": -1, "input_data_size": 18583969}
>2018-07-11 09:02:33.045289 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 870] Generated table #10432: 3439 keys, 16655889 bytes
>2018-07-11 09:02:33.045339 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299753045323, "cf_name": "default", "job": 870, "event": "table_file_creation", "file_number": 10432, "file_size": 16655889, "table_properties": {"data_size": 16529261, "index_size": 72225, "filter_size": 53476, "raw_key_size": 69630, "raw_average_key_size": 20, "raw_value_size": 16429642, "raw_average_value_size": 4777, "num_data_blocks": 2103, "num_entries": 3439, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:02:33.086805 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 870] Compacted 1@0 + 1@1 files to L1 => 16655889 bytes
>2018-07-11 09:02:33.120315 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:02:33.120247) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.1 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(3.2, 14.6) out(15.9), read-write-amplify(10.6) write-amplify(5.0) OK, records in: 3949, records dropped: 510
>
>2018-07-11 09:02:33.120324 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:02:33.120269) EVENT_LOG_v1 {"time_micros": 1531299753120259, "job": 870, "event": "compaction_finished", "compaction_time_micros": 241139, "output_level": 1, "num_output_files": 1, "total_output_size": 16655889, "num_input_records": 3949, "num_output_records": 3439, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:02:33.120987 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299753120983, "job": 870, "event": "table_file_deletion", "file_number": 10431}
>2018-07-11 09:02:33.123917 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299753123911, "job": 870, "event": "table_file_deletion", "file_number": 10429}
>2018-07-11 09:03:02.590609 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:03:35.693237 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10433. Immutable memtables: 0.
>
>2018-07-11 09:03:35.693279 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 871] Syncing log #10430
>2018-07-11 09:03:35.709736 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.693270) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:03:35.709748 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 871] Flushing memtable with next log file: 10433
>
>2018-07-11 09:03:35.709773 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299815709765, "job": 871, "event": "flush_started", "num_memtables": 1, "num_entries": 472, "num_deletes": 250, "memory_usage": 484568}
>2018-07-11 09:03:35.709776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 871] Level-0 flush table #10434: started
>2018-07-11 09:03:35.743574 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299815743561, "cf_name": "default", "job": 871, "event": "table_file_creation", "file_number": 10434, "file_size": 288246, "table_properties": {"data_size": 284513, "index_size": 1273, "filter_size": 1541, "raw_key_size": 6430, "raw_average_key_size": 21, "raw_value_size": 279625, "raw_average_value_size": 951, "num_data_blocks": 40, "num_entries": 294, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:03:35.743584 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 871] Level-0 flush table #10434: 288246 bytes OK
>2018-07-11 09:03:35.801962 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.760136) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10434 started
>2018-07-11 09:03:35.801970 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.801927) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10434: memtable #1 done
>2018-07-11 09:03:35.801972 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.801941) EVENT_LOG_v1 {"time_micros": 1531299815801935, "job": 871, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:03:35.801974 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.801953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:03:35.801994 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 871] Try to delete WAL files size 473754, prev total WAL file size 473754, number of live WAL files 2.
>
>2018-07-11 09:03:35.802169 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:03:35.802228 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:03:35.802220) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:03:35.802234 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 872] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:03:35.802240 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 871 Base level 0, inputs: [10434(281KB)], [10432(15MB)]
>
>2018-07-11 09:03:35.802253 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299815802244, "job": 872, "event": "compaction_started", "files_L0": [10434], "files_L1": [10432], "score": -1, "input_data_size": 16944135}
>2018-07-11 09:03:35.986017 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 872] Generated table #10435: 3252 keys, 15404029 bytes
>2018-07-11 09:03:35.986058 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299815986031, "cf_name": "default", "job": 872, "event": "table_file_creation", "file_number": 10435, "file_size": 15404029, "table_properties": {"data_size": 15288096, "index_size": 65637, "filter_size": 49369, "raw_key_size": 65452, "raw_average_key_size": 20, "raw_value_size": 15195834, "raw_average_value_size": 4672, "num_data_blocks": 1916, "num_entries": 3252, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:03:36.019198 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 872] Compacted 1@0 + 1@1 files to L1 => 15404029 bytes
>2018-07-11 09:03:36.061055 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:03:36.060985) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.1 rd, 71.0 wr, level 1, files in(1, 1) out(1) MB in(0.3, 15.9) out(14.7), read-write-amplify(112.2) write-amplify(53.4) OK, records in: 3733, records dropped: 481
>
>2018-07-11 09:03:36.061064 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:03:36.061007) EVENT_LOG_v1 {"time_micros": 1531299816060997, "job": 872, "event": "compaction_finished", "compaction_time_micros": 216931, "output_level": 1, "num_output_files": 1, "total_output_size": 15404029, "num_input_records": 3733, "num_output_records": 3252, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:03:36.061210 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299816061206, "job": 872, "event": "table_file_deletion", "file_number": 10434}
>2018-07-11 09:03:36.064125 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531299816064123, "job": 872, "event": "table_file_deletion", "file_number": 10432}
>2018-07-11 09:03:36.064245 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:04:02.622856 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:05:02.643592 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:06:02.673674 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:06:34.037375 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:06:34.037399 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3250271774' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:06:34.168846 7f29149e4700  1 mon.magna102@0(leader).log v20407 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20407)
>2018-07-11 09:07:02.695810 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:08:02.726033 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:09:02.748102 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:10:02.785507 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:10:46.060075 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:10:46.060098 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/407895213' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:10:46.173192 7f29149e4700  1 mon.magna102@0(leader).log v20408 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20408)
>2018-07-11 09:10:47.307118 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10436. Immutable memtables: 0.
>
>2018-07-11 09:10:47.307163 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 873] Syncing log #10433
>2018-07-11 09:10:47.323641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.307154) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:10:47.323654 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 873] Flushing memtable with next log file: 10436
>
>2018-07-11 09:10:47.323670 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247323661, "job": 873, "event": "flush_started", "num_memtables": 1, "num_entries": 1784, "num_deletes": 251, "memory_usage": 3285672}
>2018-07-11 09:10:47.323673 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 873] Level-0 flush table #10437: started
>2018-07-11 09:10:47.390851 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247390834, "cf_name": "default", "job": 873, "event": "table_file_creation", "file_number": 10437, "file_size": 3137576, "table_properties": {"data_size": 3111473, "index_size": 14523, "filter_size": 10656, "raw_key_size": 14441, "raw_average_key_size": 20, "raw_value_size": 3091358, "raw_average_value_size": 4428, "num_data_blocks": 438, "num_entries": 698, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:10:47.390865 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 873] Level-0 flush table #10437: 3137576 bytes OK
>2018-07-11 09:10:47.440881 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.407350) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10437 started
>2018-07-11 09:10:47.440889 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.440831) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10437: memtable #1 done
>2018-07-11 09:10:47.440892 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.440847) EVENT_LOG_v1 {"time_micros": 1531300247440840, "job": 873, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:10:47.440894 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.440867) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:10:47.440901 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 873] Try to delete WAL files size 3251253, prev total WAL file size 3251253, number of live WAL files 2.
>
>2018-07-11 09:10:47.441741 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:10:47.441927 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 09:10:47.441934 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 62274.3 total, 972.8 interval
>Cumulative writes: 85K writes, 404K keys, 85K commit groups, 1.0 writes per commit group, ingest: 0.67 GB, 0.01 MB/s
>Cumulative WAL: 85K writes, 85K syncs, 1.00 writes per sync, written: 0.67 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4520 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.12 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.99 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        36       437    0.081       0      0
>  L1      1/1   14.69 MB   0.0      6.9     0.6      6.3       6.3     -0.0       0.0  10.7     72.5     66.3        97       435    0.223   1550K   185K
> Sum      2/2   17.68 MB   0.0      6.9     0.6      6.3       6.9      0.6       0.0  11.7     53.1     53.1       132       872    0.152   1550K   185K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.2     56.8     56.7         1         8    0.149     15K   1993
>Uptime(secs): 62274.3 total, 62274.3 interval
>Flush(GB): cumulative 0.588, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.86 GB write, 0.11 MB/s write, 6.86 GB read, 0.11 MB/s read, 132.4 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.99 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        36       437    0.081       0      0
>  L1      1/1   14.69 MB   0.0      6.9     0.6      6.3       6.3     -0.0       0.0  10.7     72.5     66.3        97       435    0.223   1550K   185K
> Sum      2/2   17.68 MB   0.0      6.9     0.6      6.3       6.9      0.6       0.0  11.7     53.1     53.1       132       872    0.152   1550K   185K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 62274.3 total, 62274.3 interval
>Flush(GB): cumulative 0.588, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.86 GB write, 0.11 MB/s write, 6.86 GB read, 0.11 MB/s read, 132.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 09:10:47.441956 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.441949) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:10:47.441961 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 874] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:10:47.441966 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 873 Base level 0, inputs: [10437(3064KB)], [10435(14MB)]
>
>2018-07-11 09:10:47.441979 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247441969, "job": 874, "event": "compaction_started", "files_L0": [10437], "files_L1": [10435], "score": -1, "input_data_size": 18541605}
>2018-07-11 09:10:47.569324 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:10:47.569355 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2739503442' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:10:47.633360 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 874] Generated table #10438: 3440 keys, 16646258 bytes
>2018-07-11 09:10:47.633392 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247633378, "cf_name": "default", "job": 874, "event": "table_file_creation", "file_number": 10438, "file_size": 16646258, "table_properties": {"data_size": 16519681, "index_size": 72194, "filter_size": 53456, "raw_key_size": 69646, "raw_average_key_size": 20, "raw_value_size": 16420069, "raw_average_value_size": 4773, "num_data_blocks": 2102, "num_entries": 3440, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:10:47.666502 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 874] Compacted 1@0 + 1@1 files to L1 => 16646258 bytes
>2018-07-11 09:10:47.708360 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.708289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.6 rd, 74.1 wr, level 1, files in(1, 1) out(1) MB in(3.0, 14.7) out(15.9), read-write-amplify(11.2) write-amplify(5.3) OK, records in: 3950, records dropped: 510
>
>2018-07-11 09:10:47.708369 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:10:47.708310) EVENT_LOG_v1 {"time_micros": 1531300247708301, "job": 874, "event": "compaction_finished", "compaction_time_micros": 224506, "output_level": 1, "num_output_files": 1, "total_output_size": 16646258, "num_input_records": 3950, "num_output_records": 3440, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:10:47.708988 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247708984, "job": 874, "event": "table_file_deletion", "file_number": 10437}
>2018-07-11 09:10:47.711963 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300247711960, "job": 874, "event": "table_file_deletion", "file_number": 10435}
>2018-07-11 09:10:48.375599 7f29149e4700  1 mon.magna102@0(leader).log v20409 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20409)
>2018-07-11 09:11:02.801650 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:11:14.348554 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd dump"} v 0) v1
>2018-07-11 09:11:14.348582 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/503989690' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch
>2018-07-11 09:11:14.470614 7f29149e4700  1 mon.magna102@0(leader).log v20410 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20410)
>2018-07-11 09:11:21.913143 7f29149e4700  1 mon.magna102@0(leader).log v20411 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20411)
>2018-07-11 09:11:37.922995 7f29149e4700  1 mon.magna102@0(leader).log v20412 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20412)
>2018-07-11 09:11:40.064973 7f29149e4700  1 mon.magna102@0(leader).log v20413 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20413)
>2018-07-11 09:11:55.900274 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10439. Immutable memtables: 0.
>
>2018-07-11 09:11:55.900317 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 875] Syncing log #10436
>2018-07-11 09:11:55.916652 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:11:55.900307) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:11:55.916663 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 875] Flushing memtable with next log file: 10439
>
>2018-07-11 09:11:55.916689 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300315916681, "job": 875, "event": "flush_started", "num_memtables": 1, "num_entries": 533, "num_deletes": 250, "memory_usage": 607632}
>2018-07-11 09:11:55.916693 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 875] Level-0 flush table #10440: started
>2018-07-11 09:11:55.958837 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300315958823, "cf_name": "default", "job": 875, "event": "table_file_creation", "file_number": 10440, "file_size": 388329, "table_properties": {"data_size": 383886, "index_size": 1664, "filter_size": 1860, "raw_key_size": 6846, "raw_average_key_size": 21, "raw_value_size": 378408, "raw_average_value_size": 1205, "num_data_blocks": 52, "num_entries": 314, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:11:55.958847 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 875] Level-0 flush table #10440: 388329 bytes OK
>2018-07-11 09:11:56.008875 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:11:55.975381) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10440 started
>2018-07-11 09:11:56.008883 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.008839) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10440: memtable #1 done
>2018-07-11 09:11:56.008885 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.008853) EVENT_LOG_v1 {"time_micros": 1531300316008847, "job": 875, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:11:56.008901 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.008865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:11:56.008907 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 875] Try to delete WAL files size 595915, prev total WAL file size 595915, number of live WAL files 2.
>
>2018-07-11 09:11:56.009195 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:11:56.009246 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.009239) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:11:56.009255 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 876] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:11:56.009262 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 875 Base level 0, inputs: [10440(379KB)], [10438(15MB)]
>
>2018-07-11 09:11:56.009276 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300316009266, "job": 876, "event": "compaction_started", "files_L0": [10440], "files_L1": [10438], "score": -1, "input_data_size": 17034587}
>2018-07-11 09:11:56.185826 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 876] Generated table #10441: 3274 keys, 15515138 bytes
>2018-07-11 09:11:56.185875 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300316185848, "cf_name": "default", "job": 876, "event": "table_file_creation", "file_number": 10441, "file_size": 15515138, "table_properties": {"data_size": 15398322, "index_size": 66152, "filter_size": 49737, "raw_key_size": 65899, "raw_average_key_size": 20, "raw_value_size": 15305390, "raw_average_value_size": 4674, "num_data_blocks": 1931, "num_entries": 3274, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:11:56.218958 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 876] Compacted 1@0 + 1@1 files to L1 => 15515138 bytes
>2018-07-11 09:11:56.252520 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.252448) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.2 rd, 74.0 wr, level 1, files in(1, 1) out(1) MB in(0.4, 15.9) out(14.8), read-write-amplify(83.8) write-amplify(40.0) OK, records in: 3754, records dropped: 480
>
>2018-07-11 09:11:56.252528 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:11:56.252473) EVENT_LOG_v1 {"time_micros": 1531300316252461, "job": 876, "event": "compaction_finished", "compaction_time_micros": 209665, "output_level": 1, "num_output_files": 1, "total_output_size": 15515138, "num_input_records": 3754, "num_output_records": 3274, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:11:56.252681 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300316252677, "job": 876, "event": "table_file_deletion", "file_number": 10440}
>2018-07-11 09:11:56.255636 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300316255633, "job": 876, "event": "table_file_deletion", "file_number": 10438}
>2018-07-11 09:11:56.255752 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:12:02.828315 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:12:28.838464 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush dump"} v 0) v1
>2018-07-11 09:12:28.838499 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1288104196' entity='client.admin' cmd=[{"prefix": "osd crush dump"}]: dispatch
>2018-07-11 09:12:28.955598 7f29149e4700  1 mon.magna102@0(leader).log v20414 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20414)
>2018-07-11 09:13:01.119538 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd erasure-code-profile get", "name": "newprofile"} v 0) v1
>2018-07-11 09:13:01.119573 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2808079813' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile get", "name": "newprofile"}]: dispatch
>2018-07-11 09:13:01.951013 7f29149e4700  1 mon.magna102@0(leader).log v20415 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20415)
>2018-07-11 09:13:02.828485 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:14:02.828668 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:15:02.828847 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:16:02.829016 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:17:02.829200 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:18:02.829378 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:18:51.105882 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10442. Immutable memtables: 0.
>
>2018-07-11 09:18:51.105927 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 877] Syncing log #10439
>2018-07-11 09:18:51.122396 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.105917) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:18:51.122408 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 877] Flushing memtable with next log file: 10442
>
>2018-07-11 09:18:51.122434 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731122426, "job": 877, "event": "flush_started", "num_memtables": 1, "num_entries": 1728, "num_deletes": 251, "memory_usage": 3165304}
>2018-07-11 09:18:51.122438 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 877] Level-0 flush table #10443: started
>2018-07-11 09:18:51.181272 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731181258, "cf_name": "default", "job": 877, "event": "table_file_creation", "file_number": 10443, "file_size": 3022780, "table_properties": {"data_size": 2997586, "index_size": 13982, "filter_size": 10288, "raw_key_size": 14105, "raw_average_key_size": 20, "raw_value_size": 2978076, "raw_average_value_size": 4366, "num_data_blocks": 422, "num_entries": 682, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:18:51.181283 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 877] Level-0 flush table #10443: 3022780 bytes OK
>2018-07-11 09:18:51.231302 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.197792) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10443 started
>2018-07-11 09:18:51.231311 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.231263) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10443: memtable #1 done
>2018-07-11 09:18:51.231313 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.231278) EVENT_LOG_v1 {"time_micros": 1531300731231272, "job": 877, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:18:51.231315 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.231292) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:18:51.231321 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 877] Try to delete WAL files size 3132179, prev total WAL file size 3132179, number of live WAL files 2.
>
>2018-07-11 09:18:51.232094 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:18:51.232157 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.232150) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:18:51.232164 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 878] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:18:51.232170 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 877 Base level 0, inputs: [10443(2951KB)], [10441(14MB)]
>
>2018-07-11 09:18:51.232192 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731232173, "job": 878, "event": "compaction_started", "files_L0": [10443], "files_L1": [10441], "score": -1, "input_data_size": 18537918}
>2018-07-11 09:18:51.431985 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 878] Generated table #10444: 3446 keys, 16639809 bytes
>2018-07-11 09:18:51.432034 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731432007, "cf_name": "default", "job": 878, "event": "table_file_creation", "file_number": 10444, "file_size": 16639809, "table_properties": {"data_size": 16513229, "index_size": 72198, "filter_size": 53455, "raw_key_size": 69757, "raw_average_key_size": 20, "raw_value_size": 16413536, "raw_average_value_size": 4763, "num_data_blocks": 2102, "num_entries": 3446, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:18:51.465140 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 878] Compacted 1@0 + 1@1 files to L1 => 16639809 bytes
>2018-07-11 09:18:51.498701 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.498634) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.6 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(2.9, 14.8) out(15.9), read-write-amplify(11.6) write-amplify(5.5) OK, records in: 3956, records dropped: 510
>
>2018-07-11 09:18:51.498710 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:18:51.498654) EVENT_LOG_v1 {"time_micros": 1531300731498645, "job": 878, "event": "compaction_finished", "compaction_time_micros": 232933, "output_level": 1, "num_output_files": 1, "total_output_size": 16639809, "num_input_records": 3956, "num_output_records": 3446, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:18:51.499421 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731499417, "job": 878, "event": "table_file_deletion", "file_number": 10443}
>2018-07-11 09:18:51.502415 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300731502412, "job": 878, "event": "table_file_deletion", "file_number": 10441}
>2018-07-11 09:19:02.829531 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:19:45.699056 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 09:19:45.699082 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2664260794' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 09:19:46.137599 7f29149e4700  1 mon.magna102@0(leader).log v20416 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20416)
>2018-07-11 09:20:02.829712 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:20:16.157424 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10445. Immutable memtables: 0.
>
>2018-07-11 09:20:16.157468 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 879] Syncing log #10442
>2018-07-11 09:20:16.173885 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.157458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:20:16.173896 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 879] Flushing memtable with next log file: 10445
>
>2018-07-11 09:20:16.173923 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816173915, "job": 879, "event": "flush_started", "num_memtables": 1, "num_entries": 564, "num_deletes": 250, "memory_usage": 666688}
>2018-07-11 09:20:16.173926 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 879] Level-0 flush table #10446: started
>2018-07-11 09:20:16.224425 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816224411, "cf_name": "default", "job": 879, "event": "table_file_creation", "file_number": 10446, "file_size": 387986, "table_properties": {"data_size": 383556, "index_size": 1654, "filter_size": 1857, "raw_key_size": 6852, "raw_average_key_size": 21, "raw_value_size": 378062, "raw_average_value_size": 1204, "num_data_blocks": 52, "num_entries": 314, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:20:16.224435 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 879] Level-0 flush table #10446: 387986 bytes OK
>2018-07-11 09:20:16.274455 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.240962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10446 started
>2018-07-11 09:20:16.274463 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.274419) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10446: memtable #1 done
>2018-07-11 09:20:16.274466 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.274434) EVENT_LOG_v1 {"time_micros": 1531300816274427, "job": 879, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:20:16.274480 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.274445) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:20:16.274487 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 879] Try to delete WAL files size 654407, prev total WAL file size 654407, number of live WAL files 2.
>
>2018-07-11 09:20:16.274773 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:20:16.274825 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.274818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:20:16.274834 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 880] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:20:16.274840 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 879 Base level 0, inputs: [10446(378KB)], [10444(15MB)]
>
>2018-07-11 09:20:16.274856 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816274844, "job": 880, "event": "compaction_started", "files_L0": [10446], "files_L1": [10444], "score": -1, "input_data_size": 17027795}
>2018-07-11 09:20:16.458516 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 880] Generated table #10447: 3286 keys, 15562789 bytes
>2018-07-11 09:20:16.458565 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816458538, "cf_name": "default", "job": 880, "event": "table_file_creation", "file_number": 10447, "file_size": 15562789, "table_properties": {"data_size": 15445534, "index_size": 66424, "filter_size": 49904, "raw_key_size": 66138, "raw_average_key_size": 20, "raw_value_size": 15352247, "raw_average_value_size": 4672, "num_data_blocks": 1939, "num_entries": 3286, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:20:16.491695 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 880] Compacted 1@0 + 1@1 files to L1 => 15562789 bytes
>2018-07-11 09:20:16.533552 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.533481) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.5 rd, 71.8 wr, level 1, files in(1, 1) out(1) MB in(0.4, 15.9) out(14.8), read-write-amplify(84.0) write-amplify(40.1) OK, records in: 3760, records dropped: 474
>
>2018-07-11 09:20:16.533561 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:20:16.533503) EVENT_LOG_v1 {"time_micros": 1531300816533493, "job": 880, "event": "compaction_finished", "compaction_time_micros": 216822, "output_level": 1, "num_output_files": 1, "total_output_size": 15562789, "num_input_records": 3760, "num_output_records": 3286, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:20:16.533715 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816533711, "job": 880, "event": "table_file_deletion", "file_number": 10446}
>2018-07-11 09:20:16.536738 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531300816536735, "job": 880, "event": "table_file_deletion", "file_number": 10444}
>2018-07-11 09:20:16.536852 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:21:02.829880 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:21:08.162349 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush dump"} v 0) v1
>2018-07-11 09:21:08.162378 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2167409604' entity='client.admin' cmd=[{"prefix": "osd crush dump"}]: dispatch
>2018-07-11 09:21:08.289522 7f29149e4700  1 mon.magna102@0(leader).log v20417 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20417)
>2018-07-11 09:21:54.592016 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:21:54.592040 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2797907142' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:21:54.711599 7f29149e4700  1 mon.magna102@0(leader).log v20418 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20418)
>2018-07-11 09:22:02.830030 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:22:38.215019 7f29149e4700  1 mon.magna102@0(leader).log v20419 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20419)
>2018-07-11 09:22:50.242283 7f29149e4700  1 mon.magna102@0(leader).log v20420 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20420)
>2018-07-11 09:23:02.830204 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:24:02.830364 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:25:02.830520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:25:08.833141 7f29149e4700  1 mon.magna102@0(leader).log v20421 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20421)
>2018-07-11 09:25:28.834717 7f29149e4700  1 mon.magna102@0(leader).log v20422 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20422)
>2018-07-11 09:25:38.843887 7f29149e4700  1 mon.magna102@0(leader).log v20423 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20423)
>2018-07-11 09:25:57.164530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:25:57.305268 7f29149e4700  1 mon.magna102@0(leader).log v20424 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20424)
>2018-07-11 09:26:02.830691 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:26:49.326108 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10448. Immutable memtables: 0.
>
>2018-07-11 09:26:49.326232 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 881] Syncing log #10445
>2018-07-11 09:26:49.343623 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.326222) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:26:49.343634 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 881] Flushing memtable with next log file: 10448
>
>2018-07-11 09:26:49.343661 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209343653, "job": 881, "event": "flush_started", "num_memtables": 1, "num_entries": 1699, "num_deletes": 251, "memory_usage": 3105368}
>2018-07-11 09:26:49.343665 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 881] Level-0 flush table #10449: started
>2018-07-11 09:26:49.410848 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209410833, "cf_name": "default", "job": 881, "event": "table_file_creation", "file_number": 10449, "file_size": 2963500, "table_properties": {"data_size": 2938829, "index_size": 13650, "filter_size": 10097, "raw_key_size": 14009, "raw_average_key_size": 20, "raw_value_size": 2919622, "raw_average_value_size": 4306, "num_data_blocks": 412, "num_entries": 678, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:26:49.410859 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 881] Level-0 flush table #10449: 2963500 bytes OK
>2018-07-11 09:26:49.460851 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.427358) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10449 started
>2018-07-11 09:26:49.460859 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.460813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10449: memtable #1 done
>2018-07-11 09:26:49.460862 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.460827) EVENT_LOG_v1 {"time_micros": 1531301209460821, "job": 881, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:26:49.460876 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.460841) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:26:49.460883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 881] Try to delete WAL files size 3071805, prev total WAL file size 3071805, number of live WAL files 2.
>
>2018-07-11 09:26:49.461707 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:26:49.461899 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 09:26:49.461906 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 63236.3 total, 962.0 interval
>Cumulative writes: 86K writes, 408K keys, 86K commit groups, 1.0 writes per commit group, ingest: 0.68 GB, 0.01 MB/s
>Cumulative WAL: 86K writes, 86K syncs, 1.00 writes per sync, written: 0.68 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4524 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.11 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.83 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       441    0.081       0      0
>  L1      1/1   14.84 MB   0.0      6.9     0.6      6.3       6.3     -0.0       0.0  10.7     72.6     66.4        98       439    0.223   1565K   187K
> Sum      2/2   17.67 MB   0.0      6.9     0.6      6.3       6.9      0.6       0.0  11.7     53.1     53.1       134       880    0.152   1565K   187K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.5     58.0     58.0         1         8    0.146     15K   1974
>Uptime(secs): 63236.3 total, 63236.3 interval
>Flush(GB): cumulative 0.594, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.93 GB write, 0.11 MB/s write, 6.93 GB read, 0.11 MB/s read, 133.6 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.83 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       441    0.081       0      0
>  L1      1/1   14.84 MB   0.0      6.9     0.6      6.3       6.3     -0.0       0.0  10.7     72.6     66.4        98       439    0.223   1565K   187K
> Sum      2/2   17.67 MB   0.0      6.9     0.6      6.3       6.9      0.6       0.0  11.7     53.1     53.1       134       880    0.152   1565K   187K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 63236.3 total, 63236.3 interval
>Flush(GB): cumulative 0.594, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.93 GB write, 0.11 MB/s write, 6.93 GB read, 0.11 MB/s read, 133.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 09:26:49.461927 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.461920) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:26:49.461932 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 882] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:26:49.461937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 881 Base level 0, inputs: [10449(2894KB)], [10447(14MB)]
>
>2018-07-11 09:26:49.461949 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209461940, "job": 882, "event": "compaction_started", "files_L0": [10449], "files_L1": [10447], "score": -1, "input_data_size": 18526289}
>2018-07-11 09:26:49.661571 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 882] Generated table #10450: 3454 keys, 16635318 bytes
>2018-07-11 09:26:49.661620 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209661592, "cf_name": "default", "job": 882, "event": "table_file_creation", "file_number": 10450, "file_size": 16635318, "table_properties": {"data_size": 16508850, "index_size": 72101, "filter_size": 53440, "raw_key_size": 69900, "raw_average_key_size": 20, "raw_value_size": 16409115, "raw_average_value_size": 4750, "num_data_blocks": 2099, "num_entries": 3454, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:26:49.694757 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 882] Compacted 1@0 + 1@1 files to L1 => 16635318 bytes
>2018-07-11 09:26:49.728267 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.728201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.6 rd, 71.5 wr, level 1, files in(1, 1) out(1) MB in(2.8, 14.8) out(15.9), read-write-amplify(11.9) write-amplify(5.6) OK, records in: 3964, records dropped: 510
>
>2018-07-11 09:26:49.728275 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:26:49.728222) EVENT_LOG_v1 {"time_micros": 1531301209728212, "job": 882, "event": "compaction_finished", "compaction_time_micros": 232791, "output_level": 1, "num_output_files": 1, "total_output_size": 16635318, "num_input_records": 3964, "num_output_records": 3454, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:26:49.728903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209728899, "job": 882, "event": "table_file_deletion", "file_number": 10449}
>2018-07-11 09:26:49.731973 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301209731970, "job": 882, "event": "table_file_deletion", "file_number": 10447}
>2018-07-11 09:27:02.830856 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:27:07.195270 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:27:07.346791 7f29149e4700  1 mon.magna102@0(leader).log v20425 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20425)
>2018-07-11 09:27:23.202024 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:27:23.348140 7f29149e4700  1 mon.magna102@0(leader).log v20426 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20426)
>2018-07-11 09:27:31.240191 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:27:31.365573 7f29149e4700  1 mon.magna102@0(leader).log v20427 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20427)
>2018-07-11 09:28:02.831031 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:28:13.035547 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:13.035570 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3428636497' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:13.152967 7f29149e4700  1 mon.magna102@0(leader).log v20428 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20428)
>2018-07-11 09:28:15.905978 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:15.906001 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/330186604' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:16.353482 7f29149e4700  1 mon.magna102@0(leader).log v20429 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20429)
>2018-07-11 09:28:17.222945 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:17.222969 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2016386344' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:17.420405 7f29149e4700  1 mon.magna102@0(leader).log v20430 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20430)
>2018-07-11 09:28:18.540896 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:18.540919 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3673110658' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:18.662484 7f29149e4700  1 mon.magna102@0(leader).log v20431 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20431)
>2018-07-11 09:28:19.916177 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:19.916209 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3100474620' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:20.804728 7f29149e4700  1 mon.magna102@0(leader).log v20432 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20432)
>2018-07-11 09:28:21.435818 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:21.435841 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1586048772' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:21.871570 7f29149e4700  1 mon.magna102@0(leader).log v20433 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20433)
>2018-07-11 09:28:22.764597 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:22.764620 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2902245550' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:23.005317 7f29149e4700  1 mon.magna102@0(leader).log v20434 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20434)
>2018-07-11 09:28:23.992046 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:23.992070 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3822749107' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:24.138969 7f29149e4700  1 mon.magna102@0(leader).log v20435 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20435)
>2018-07-11 09:28:25.311334 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:25.311358 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1552955974' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:26.364354 7f29149e4700  1 mon.magna102@0(leader).log v20436 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20436)
>2018-07-11 09:28:26.671455 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:28:26.671478 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4142168766' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:28:27.431177 7f29149e4700  1 mon.magna102@0(leader).log v20437 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20437)
>2018-07-11 09:28:36.350332 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10451. Immutable memtables: 0.
>
>2018-07-11 09:28:36.350377 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 883] Syncing log #10448
>2018-07-11 09:28:36.366773 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.350367) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:28:36.366784 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 883] Flushing memtable with next log file: 10451
>
>2018-07-11 09:28:36.366810 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316366790, "job": 883, "event": "flush_started", "num_memtables": 1, "num_entries": 730, "num_deletes": 250, "memory_usage": 1030840}
>2018-07-11 09:28:36.366813 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 883] Level-0 flush table #10452: started
>2018-07-11 09:28:36.417320 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316417306, "cf_name": "default", "job": 883, "event": "table_file_creation", "file_number": 10452, "file_size": 659108, "table_properties": {"data_size": 652664, "index_size": 2806, "filter_size": 2719, "raw_key_size": 7714, "raw_average_key_size": 21, "raw_value_size": 645763, "raw_average_value_size": 1808, "num_data_blocks": 87, "num_entries": 357, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:28:36.417330 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 883] Level-0 flush table #10452: 659108 bytes OK
>2018-07-11 09:28:36.467326 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.433823) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10452 started
>2018-07-11 09:28:36.467334 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.467290) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10452: memtable #1 done
>2018-07-11 09:28:36.467336 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.467304) EVENT_LOG_v1 {"time_micros": 1531301316467298, "job": 883, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:28:36.467350 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.467316) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:28:36.467357 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 883] Try to delete WAL files size 1015648, prev total WAL file size 1015648, number of live WAL files 2.
>
>2018-07-11 09:28:36.467694 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:28:36.467746 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.467739) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:28:36.467756 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 884] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:28:36.467762 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 883 Base level 0, inputs: [10452(643KB)], [10450(15MB)]
>
>2018-07-11 09:28:36.467775 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316467766, "job": 884, "event": "compaction_started", "files_L0": [10452], "files_L1": [10450], "score": -1, "input_data_size": 17294426}
>2018-07-11 09:28:36.651376 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 884] Generated table #10453: 3350 keys, 15902794 bytes
>2018-07-11 09:28:36.651427 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316651409, "cf_name": "default", "job": 884, "event": "table_file_creation", "file_number": 10453, "file_size": 15902794, "table_properties": {"data_size": 15782934, "index_size": 67928, "filter_size": 51005, "raw_key_size": 67439, "raw_average_key_size": 20, "raw_value_size": 15687678, "raw_average_value_size": 4682, "num_data_blocks": 1983, "num_entries": 3350, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:28:36.684549 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 884] Compacted 1@0 + 1@1 files to L1 => 15902794 bytes
>2018-07-11 09:28:36.718080 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.718010) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.8 rd, 73.4 wr, level 1, files in(1, 1) out(1) MB in(0.6, 15.9) out(15.2), read-write-amplify(50.4) write-amplify(24.1) OK, records in: 3811, records dropped: 461
>
>2018-07-11 09:28:36.718089 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:28:36.718032) EVENT_LOG_v1 {"time_micros": 1531301316718022, "job": 884, "event": "compaction_finished", "compaction_time_micros": 216757, "output_level": 1, "num_output_files": 1, "total_output_size": 15902794, "num_input_records": 3811, "num_output_records": 3350, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:28:36.718302 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316718298, "job": 884, "event": "table_file_deletion", "file_number": 10452}
>2018-07-11 09:28:36.721289 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301316721286, "job": 884, "event": "table_file_deletion", "file_number": 10450}
>2018-07-11 09:28:36.721330 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:28:37.350058 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:28:37.519053 7f29149e4700  1 mon.magna102@0(leader).log v20438 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20438)
>2018-07-11 09:28:53.243466 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:28:53.395307 7f29149e4700  1 mon.magna102@0(leader).log v20439 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20439)
>2018-07-11 09:29:01.317553 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:29:02.379343 7f29149e4700  1 mon.magna102@0(leader).log v20440 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20440)
>2018-07-11 09:29:02.831222 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:30:02.831403 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:30:07.276244 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:30:07.427145 7f29149e4700  1 mon.magna102@0(leader).log v20441 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20441)
>2018-07-11 09:30:15.687728 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:15.687752 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/954880914' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:16.411190 7f29149e4700  1 mon.magna102@0(leader).log v20442 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20442)
>2018-07-11 09:30:17.873904 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:17.873927 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2254647672' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:18.544777 7f29149e4700  1 mon.magna102@0(leader).log v20443 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20443)
>2018-07-11 09:30:19.378028 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:19.378051 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4279672608' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:19.611706 7f29149e4700  1 mon.magna102@0(leader).log v20444 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20444)
>2018-07-11 09:30:20.739965 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:20.739989 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3410936034' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:20.853775 7f29149e4700  1 mon.magna102@0(leader).log v20445 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20445)
>2018-07-11 09:30:21.953791 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:21.953814 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3420614826' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:23.004035 7f29149e4700  1 mon.magna102@0(leader).log v20446 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20446)
>2018-07-11 09:30:23.291939 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:23.291964 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1873560466' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:24.037538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:30:24.070977 7f29149e4700  1 mon.magna102@0(leader).log v20447 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20447)
>2018-07-11 09:30:24.546945 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:24.546969 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2799151543' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:25.156034 7f29149e4700  1 mon.magna102@0(leader).log v20448 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20448)
>2018-07-11 09:30:26.607634 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:26.607657 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/616938823' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:27.381468 7f29149e4700  1 mon.magna102@0(leader).log v20449 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20449)
>2018-07-11 09:30:28.055639 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:28.055662 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3339773647' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:28.448355 7f29149e4700  1 mon.magna102@0(leader).log v20450 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20450)
>2018-07-11 09:30:29.367739 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:29.367762 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2115641139' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:29.515149 7f29149e4700  1 mon.magna102@0(leader).log v20451 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20451)
>2018-07-11 09:30:30.564067 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:30.564091 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3351627057' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:30.648919 7f29149e4700  1 mon.magna102@0(leader).log v20452 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20452)
>2018-07-11 09:30:31.320900 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:30:31.715704 7f29149e4700  1 mon.magna102@0(leader).log v20453 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20453)
>2018-07-11 09:30:31.869139 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:31.869162 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2163178393' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:32.857967 7f29149e4700  1 mon.magna102@0(leader).log v20454 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20454)
>2018-07-11 09:30:33.087147 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:33.087171 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1923425073' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:33.991609 7f29149e4700  1 mon.magna102@0(leader).log v20455 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20455)
>2018-07-11 09:30:34.283414 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:34.283437 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3068984656' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:35.058567 7f29149e4700  1 mon.magna102@0(leader).log v20456 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20456)
>2018-07-11 09:30:35.413176 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:35.413209 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4258390780' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:36.133768 7f29149e4700  1 mon.magna102@0(leader).log v20457 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20457)
>2018-07-11 09:30:36.571510 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:30:36.571533 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2644109936' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:30:37.275994 7f29149e4700  1 mon.magna102@0(leader).log v20458 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20458)
>2018-07-11 09:31:02.831565 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:31:37.317706 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 09:31:37.464986 7f29149e4700  1 mon.magna102@0(leader).log v20459 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20459)
>2018-07-11 09:31:41.319464 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 09:31:41.473776 7f29149e4700  1 mon.magna102@0(leader).log v20460 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20460)
>2018-07-11 09:31:53.324512 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:31:53.474822 7f29149e4700  1 mon.magna102@0(leader).log v20461 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20461)
>2018-07-11 09:32:01.366954 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:32:01.483947 7f29149e4700  1 mon.magna102@0(leader).log v20462 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20462)
>2018-07-11 09:32:02.831736 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:32:07.509502 7f29149e4700  1 mon.magna102@0(leader).log v20463 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20463)
>2018-07-11 09:32:09.292982 7f29149e4700  1 mon.magna102@0(leader).log v20464 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20464)
>2018-07-11 09:32:31.341283 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:32:31.512886 7f29149e4700  1 mon.magna102@0(leader).log v20465 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20465)
>2018-07-11 09:32:37.343821 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:32:37.513484 7f29149e4700  1 mon.magna102@0(leader).log v20466 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20466)
>2018-07-11 09:32:46.388770 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:32:46.514192 7f29149e4700  1 mon.magna102@0(leader).log v20467 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20467)
>2018-07-11 09:32:56.389637 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 46 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:32:56.481736 7f29149e4700  1 mon.magna102@0(leader).log v20468 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20468)
>2018-07-11 09:33:01.390374 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:01.507424 7f29149e4700  1 mon.magna102@0(leader).log v20469 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20469)
>2018-07-11 09:33:02.831906 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:33:06.390575 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:06.491209 7f29149e4700  1 mon.magna102@0(leader).log v20470 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20470)
>2018-07-11 09:33:07.491221 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 58 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:33:07.641545 7f29149e4700  1 mon.magna102@0(leader).log v20471 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20471)
>2018-07-11 09:33:21.363096 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 5 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 09:33:21.510594 7f29149e4700  1 mon.magna102@0(leader).log v20472 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20472)
>2018-07-11 09:33:27.365613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:27.519529 7f29149e4700  1 mon.magna102@0(leader).log v20473 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20473)
>2018-07-11 09:33:29.503504 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10454. Immutable memtables: 0.
>
>2018-07-11 09:33:29.503549 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 885] Syncing log #10451
>2018-07-11 09:33:29.519932 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.503539) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:33:29.519944 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 885] Flushing memtable with next log file: 10454
>
>2018-07-11 09:33:29.519957 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609519950, "job": 885, "event": "flush_started", "num_memtables": 1, "num_entries": 1573, "num_deletes": 251, "memory_usage": 2948040}
>2018-07-11 09:33:29.519960 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 885] Level-0 flush table #10455: started
>2018-07-11 09:33:29.587145 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609587131, "cf_name": "default", "job": 885, "event": "table_file_creation", "file_number": 10455, "file_size": 2818458, "table_properties": {"data_size": 2795771, "index_size": 12265, "filter_size": 9498, "raw_key_size": 13617, "raw_average_key_size": 20, "raw_value_size": 2777779, "raw_average_value_size": 4196, "num_data_blocks": 370, "num_entries": 662, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:33:29.587157 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 885] Level-0 flush table #10455: 2818458 bytes OK
>2018-07-11 09:33:29.637196 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.603684) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10455 started
>2018-07-11 09:33:29.637204 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.637151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10455: memtable #1 done
>2018-07-11 09:33:29.637206 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.637165) EVENT_LOG_v1 {"time_micros": 1531301609637159, "job": 885, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:33:29.637208 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.637185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:33:29.637215 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 885] Try to delete WAL files size 2916504, prev total WAL file size 2916504, number of live WAL files 2.
>
>2018-07-11 09:33:29.637891 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:33:29.637947 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.637940) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:33:29.637956 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 886] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:33:29.637962 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 885 Base level 0, inputs: [10455(2752KB)], [10453(15MB)]
>
>2018-07-11 09:33:29.637974 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609637965, "job": 886, "event": "compaction_started", "files_L0": [10455], "files_L1": [10453], "score": -1, "input_data_size": 18721252}
>2018-07-11 09:33:29.837912 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 886] Generated table #10456: 3502 keys, 16828504 bytes
>2018-07-11 09:33:29.837956 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609837929, "cf_name": "default", "job": 886, "event": "table_file_creation", "file_number": 10456, "file_size": 16828504, "table_properties": {"data_size": 16701431, "index_size": 72207, "filter_size": 53939, "raw_key_size": 70809, "raw_average_key_size": 20, "raw_value_size": 16600940, "raw_average_value_size": 4740, "num_data_blocks": 2101, "num_entries": 3502, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:33:29.871091 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 886] Compacted 1@0 + 1@1 files to L1 => 16828504 bytes
>2018-07-11 09:33:29.904619 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.904549) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.3 rd, 72.2 wr, level 1, files in(1, 1) out(1) MB in(2.7, 15.2) out(16.0), read-write-amplify(12.6) write-amplify(6.0) OK, records in: 4012, records dropped: 510
>
>2018-07-11 09:33:29.904627 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:33:29.904571) EVENT_LOG_v1 {"time_micros": 1531301609904561, "job": 886, "event": "compaction_finished", "compaction_time_micros": 233103, "output_level": 1, "num_output_files": 1, "total_output_size": 16828504, "num_input_records": 4012, "num_output_records": 3502, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:33:29.905253 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609905249, "job": 886, "event": "table_file_deletion", "file_number": 10455}
>2018-07-11 09:33:29.908298 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301609908295, "job": 886, "event": "table_file_deletion", "file_number": 10453}
>2018-07-11 09:33:36.430246 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:36.505748 7f29149e4700  1 mon.magna102@0(leader).log v20474 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20474)
>2018-07-11 09:33:41.430498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:41.564920 7f29149e4700  1 mon.magna102@0(leader).log v20475 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20475)
>2018-07-11 09:33:46.430709 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:46.540374 7f29149e4700  1 mon.magna102@0(leader).log v20476 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20476)
>2018-07-11 09:33:51.430949 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:51.566056 7f29149e4700  1 mon.magna102@0(leader).log v20477 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20477)
>2018-07-11 09:33:56.431162 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:33:56.541500 7f29149e4700  1 mon.magna102@0(leader).log v20478 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20478)
>2018-07-11 09:34:01.431417 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:01.551718 7f29149e4700  1 mon.magna102@0(leader).log v20479 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20479)
>2018-07-11 09:34:02.832092 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:34:06.431627 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 62 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:06.518828 7f29149e4700  1 mon.magna102@0(leader).log v20480 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20480)
>2018-07-11 09:34:11.431855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 66 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:11.552845 7f29149e4700  1 mon.magna102@0(leader).log v20481 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20481)
>2018-07-11 09:34:16.432081 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 74 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:16.520054 7f29149e4700  1 mon.magna102@0(leader).log v20482 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20482)
>2018-07-11 09:34:21.432336 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 82 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:21.554050 7f29149e4700  1 mon.magna102@0(leader).log v20483 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20483)
>2018-07-11 09:34:26.432563 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 90 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:26.529480 7f29149e4700  1 mon.magna102@0(leader).log v20484 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20484)
>2018-07-11 09:34:36.432969 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 98 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:34:36.530379 7f29149e4700  1 mon.magna102@0(leader).log v20485 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20485)
>2018-07-11 09:34:37.530375 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 98 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:34:37.672423 7f29149e4700  1 mon.magna102@0(leader).log v20486 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20486)
>2018-07-11 09:34:53.403218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:34:53.557022 7f29149e4700  1 mon.magna102@0(leader).log v20487 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20487)
>2018-07-11 09:35:01.406838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:01.558988 7f29149e4700  1 mon.magna102@0(leader).log v20488 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20488)
>2018-07-11 09:35:02.832269 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:35:06.679427 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:35:06.679455 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4088600155' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:35:06.680830 7f29191ed700  1 mon.magna102@0(leader).log v20488 check_sub sending message to client.? 10.8.128.102:0/4088600155 with 1 entries (version 20488)
>2018-07-11 09:35:06.801079 7f29149e4700  1 mon.magna102@0(leader).log v20489 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20489)
>2018-07-11 09:35:06.801100 7f29149e4700  1 mon.magna102@0(leader).log v20489 check_sub sending message to client.? 10.8.128.102:0/4088600155 with 0 entries (version 20489)
>2018-07-11 09:35:07.801090 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:07.968093 7f29149e4700  1 mon.magna102@0(leader).log v20490 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20490)
>2018-07-11 09:35:07.968139 7f29149e4700  1 mon.magna102@0(leader).log v20490 check_sub sending message to client.? 10.8.128.102:0/4088600155 with 1 entries (version 20490)
>2018-07-11 09:35:10.420196 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:35:10.420224 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3706728481' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:35:10.421521 7f29191ed700  1 mon.magna102@0(leader).log v20490 check_sub sending message to client.? 10.8.128.102:0/3706728481 with 1 entries (version 20490)
>2018-07-11 09:35:10.568423 7f29149e4700  1 mon.magna102@0(leader).log v20491 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20491)
>2018-07-11 09:35:10.568455 7f29149e4700  1 mon.magna102@0(leader).log v20491 check_sub sending message to client.? 10.8.128.102:0/3706728481 with 0 entries (version 20491)
>2018-07-11 09:35:12.984257 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:35:12.984283 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3700102470' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:35:12.985633 7f29191ed700  1 mon.magna102@0(leader).log v20491 check_sub sending message to client.? 10.8.128.102:0/3700102470 with 1 entries (version 20491)
>2018-07-11 09:35:13.110424 7f29149e4700  1 mon.magna102@0(leader).log v20492 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20492)
>2018-07-11 09:35:13.110444 7f29149e4700  1 mon.magna102@0(leader).log v20492 check_sub sending message to client.? 10.8.128.102:0/3700102470 with 0 entries (version 20492)
>2018-07-11 09:35:14.984425 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:35:14.984451 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2337196029' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:35:14.985815 7f29191ed700  1 mon.magna102@0(leader).log v20492 check_sub sending message to client.? 10.8.128.102:0/2337196029 with 1 entries (version 20492)
>2018-07-11 09:35:15.277439 7f29149e4700  1 mon.magna102@0(leader).log v20493 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20493)
>2018-07-11 09:35:15.277459 7f29149e4700  1 mon.magna102@0(leader).log v20493 check_sub sending message to client.? 10.8.128.102:0/2337196029 with 0 entries (version 20493)
>2018-07-11 09:35:17.414840 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:17.586338 7f29149e4700  1 mon.magna102@0(leader).log v20494 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20494)
>2018-07-11 09:35:26.453222 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:26.578666 7f29149e4700  1 mon.magna102@0(leader).log v20495 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20495)
>2018-07-11 09:35:26.608697 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:35:26.608719 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2223367786' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:35:27.662289 7f29149e4700  1 mon.magna102@0(leader).log v20496 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20496)
>2018-07-11 09:35:31.462643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:31.579691 7f29149e4700  1 mon.magna102@0(leader).log v20497 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20497)
>2018-07-11 09:35:36.462869 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:36.555126 7f29149e4700  1 mon.magna102@0(leader).log v20498 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20498)
>2018-07-11 09:35:41.463118 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:41.580793 7f29149e4700  1 mon.magna102@0(leader).log v20499 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20499)
>2018-07-11 09:35:46.463341 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:46.556345 7f29149e4700  1 mon.magna102@0(leader).log v20500 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20500)
>2018-07-11 09:35:51.466209 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 62 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:51.583245 7f29149e4700  1 mon.magna102@0(leader).log v20501 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20501)
>2018-07-11 09:35:57.432659 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 70 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:35:57.583824 7f29149e4700  1 mon.magna102@0(leader).log v20502 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20502)
>2018-07-11 09:36:02.832444 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 09:36:06.467461 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 78 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:36:06.567977 7f29149e4700  1 mon.magna102@0(leader).log v20503 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20503)
>2018-07-11 09:36:07.567977 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 78 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 09:36:07.718326 7f29149e4700  1 mon.magna102@0(leader).log v20504 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20504)
>2018-07-11 09:36:23.443907 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:36:23.594568 7f29149e4700  1 mon.magna102@0(leader).log v20505 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20505)
>2018-07-11 09:36:31.486639 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:36:31.603693 7f29149e4700  1 mon.magna102@0(leader).log v20506 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20506)
>2018-07-11 09:36:37.450198 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:36:37.637743 7f29149e4700  1 mon.magna102@0(leader).log v20507 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20507)
>2018-07-11 09:36:47.454568 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:36:47.673171 7f29149e4700  1 mon.magna102@0(leader).log v20508 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20508)
>2018-07-11 09:36:56.498458 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 46 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:36:56.582506 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10457. Immutable memtables: 0.
>
>2018-07-11 09:36:56.582549 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 887] Syncing log #10454
>2018-07-11 09:36:56.598989 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.582539) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:36:56.599000 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 887] Flushing memtable with next log file: 10457
>
>2018-07-11 09:36:56.599026 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301816599018, "job": 887, "event": "flush_started", "num_memtables": 1, "num_entries": 1256, "num_deletes": 250, "memory_usage": 2309360}
>2018-07-11 09:36:56.599029 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 887] Level-0 flush table #10458: started
>2018-07-11 09:36:56.691400 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301816691385, "cf_name": "default", "job": 887, "event": "table_file_creation", "file_number": 10458, "file_size": 1548221, "table_properties": {"data_size": 1536024, "index_size": 5928, "filter_size": 5349, "raw_key_size": 10056, "raw_average_key_size": 21, "raw_value_size": 1525235, "raw_average_value_size": 3224, "num_data_blocks": 183, "num_entries": 473, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:36:56.691411 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 887] Level-0 flush table #10458: 1548221 bytes OK
>2018-07-11 09:36:56.733104 7f29149e4700  1 mon.magna102@0(leader).log v20509 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20509)
>2018-07-11 09:36:56.766369 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.732868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10458 started
>2018-07-11 09:36:56.766377 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.766334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10458: memtable #1 done
>2018-07-11 09:36:56.766380 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.766348) EVENT_LOG_v1 {"time_micros": 1531301816766342, "job": 887, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:36:56.766382 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.766360) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:36:56.766401 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 887] Try to delete WAL files size 2284436, prev total WAL file size 2304284, number of live WAL files 2.
>
>2018-07-11 09:36:56.766994 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:36:56.767199 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 09:36:56.767206 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 63843.6 total, 607.3 interval
>Cumulative writes: 86K writes, 412K keys, 86K commit groups, 1.0 writes per commit group, ingest: 0.69 GB, 0.01 MB/s
>Cumulative WAL: 86K writes, 86K syncs, 1.00 writes per sync, written: 0.69 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 782 writes, 3567 keys, 782 commit groups, 1.0 writes per commit group, ingest: 5.95 MB, 0.01 MB/s
>Interval WAL: 782 writes, 782 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.48 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       444    0.081       0      0
>  L1      1/1   16.05 MB   0.0      7.0     0.6      6.4       6.4      0.0       0.0  10.7     72.6     66.4        98       442    0.223   1577K   188K
> Sum      2/2   17.53 MB   0.0      7.0     0.6      6.4       7.0      0.6       0.0  11.7     53.1     53.1       135       886    0.152   1577K   188K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0  10.8     53.8     53.6         1         6    0.161     11K   1481
>Uptime(secs): 63843.6 total, 63843.6 interval
>Flush(GB): cumulative 0.599, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.98 GB write, 0.11 MB/s write, 6.98 GB read, 0.11 MB/s read, 134.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 1.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.48 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       444    0.081       0      0
>  L1      1/1   16.05 MB   0.0      7.0     0.6      6.4       6.4      0.0       0.0  10.7     72.6     66.4        98       442    0.223   1577K   188K
> Sum      2/2   17.53 MB   0.0      7.0     0.6      6.4       7.0      0.6       0.0  11.7     53.1     53.1       135       886    0.152   1577K   188K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 63843.6 total, 63843.6 interval
>Flush(GB): cumulative 0.599, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 6.98 GB write, 0.11 MB/s write, 6.98 GB read, 0.11 MB/s read, 134.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 09:36:56.767227 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:56.767220) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:36:56.767232 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 888] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:36:56.767237 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 887 Base level 0, inputs: [10458(1511KB)], [10456(16MB)]
>
>2018-07-11 09:36:56.767249 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301816767240, "job": 888, "event": "compaction_started", "files_L0": [10458], "files_L1": [10456], "score": -1, "input_data_size": 18376725}
>2018-07-11 09:36:56.983774 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 888] Generated table #10459: 3564 keys, 17322566 bytes
>2018-07-11 09:36:56.983818 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301816983791, "cf_name": "default", "job": 888, "event": "table_file_creation", "file_number": 10459, "file_size": 17322566, "table_properties": {"data_size": 17193475, "index_size": 72935, "filter_size": 55229, "raw_key_size": 71790, "raw_average_key_size": 20, "raw_value_size": 17091671, "raw_average_value_size": 4795, "num_data_blocks": 2129, "num_entries": 3564, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:36:57.016954 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 888] Compacted 1@0 + 1@1 files to L1 => 17322566 bytes
>2018-07-11 09:36:57.050477 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.050411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.6 rd, 69.4 wr, level 1, files in(1, 1) out(1) MB in(1.5, 16.0) out(16.5), read-write-amplify(23.1) write-amplify(11.2) OK, records in: 3975, records dropped: 411
>
>2018-07-11 09:36:57.050486 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.050432) EVENT_LOG_v1 {"time_micros": 1531301817050423, "job": 888, "event": "compaction_finished", "compaction_time_micros": 249688, "output_level": 1, "num_output_files": 1, "total_output_size": 17322566, "num_input_records": 3975, "num_output_records": 3564, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:36:57.050872 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817050866, "job": 888, "event": "table_file_deletion", "file_number": 10458}
>2018-07-11 09:36:57.053947 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817053944, "job": 888, "event": "table_file_deletion", "file_number": 10456}
>2018-07-11 09:36:57.054294 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10460. Immutable memtables: 0.
>
>2018-07-11 09:36:57.054333 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 889] Syncing log #10457
>2018-07-11 09:36:57.067461 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.054325) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:36:57.067472 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 889] Flushing memtable with next log file: 10460
>
>2018-07-11 09:36:57.067499 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817067491, "job": 889, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 22168}
>2018-07-11 09:36:57.067502 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 889] Level-0 flush table #10461: started
>2018-07-11 09:36:57.109557 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817109543, "cf_name": "default", "job": 889, "event": "table_file_creation", "file_number": 10461, "file_size": 20967, "table_properties": {"data_size": 19897, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 19674, "raw_average_value_size": 2459, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:36:57.109567 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 889] Level-0 flush table #10461: 20967 bytes OK
>2018-07-11 09:36:57.159635 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.126133) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10461 started
>2018-07-11 09:36:57.159643 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.159601) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10461: memtable #1 done
>2018-07-11 09:36:57.159645 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.159615) EVENT_LOG_v1 {"time_micros": 1531301817159609, "job": 889, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:36:57.159647 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.159626) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:36:57.159665 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 889] Try to delete WAL files size 19848, prev total WAL file size 19848, number of live WAL files 2.
>
>2018-07-11 09:36:57.159763 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:36:57.159809 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.159803) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:36:57.159818 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 890] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:36:57.159824 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 889 Base level 0, inputs: [10461(20KB)], [10459(16MB)]
>
>2018-07-11 09:36:57.159837 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817159827, "job": 890, "event": "compaction_started", "files_L0": [10461], "files_L1": [10459], "score": -1, "input_data_size": 17343533}
>2018-07-11 09:36:57.360384 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 890] Generated table #10462: 3567 keys, 17342415 bytes
>2018-07-11 09:36:57.360434 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817360418, "cf_name": "default", "job": 890, "event": "table_file_creation", "file_number": 10462, "file_size": 17342415, "table_properties": {"data_size": 17213198, "index_size": 73004, "filter_size": 55286, "raw_key_size": 71851, "raw_average_key_size": 20, "raw_value_size": 17111305, "raw_average_value_size": 4797, "num_data_blocks": 2131, "num_entries": 3567, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:36:57.393560 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 890] Compacted 1@0 + 1@1 files to L1 => 17342415 bytes
>2018-07-11 09:36:57.427008 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.426939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.2 rd, 74.2 wr, level 1, files in(1, 1) out(1) MB in(0.0, 16.5) out(16.5), read-write-amplify(1654.3) write-amplify(827.1) OK, records in: 3572, records dropped: 5
>
>2018-07-11 09:36:57.427017 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:36:57.426961) EVENT_LOG_v1 {"time_micros": 1531301817426951, "job": 890, "event": "compaction_finished", "compaction_time_micros": 233709, "output_level": 1, "num_output_files": 1, "total_output_size": 17342415, "num_input_records": 3572, "num_output_records": 3567, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:36:57.427095 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817427090, "job": 890, "event": "table_file_deletion", "file_number": 10461}
>2018-07-11 09:36:57.430464 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301817430461, "job": 890, "event": "table_file_deletion", "file_number": 10459}
>2018-07-11 09:37:01.540677 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:01.669921 7f29149e4700  1 mon.magna102@0(leader).log v20510 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20510)
>2018-07-11 09:37:02.832622 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 09:37:06.540942 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:06.628691 7f29149e4700  1 mon.magna102@0(leader).log v20511 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20511)
>2018-07-11 09:37:09.600314 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:37:09.600340 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1750116337' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:37:09.601593 7f29191ed700  1 mon.magna102@0(leader).log v20511 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20511)
>2018-07-11 09:37:10.629150 7f29149e4700  1 mon.magna102@0(leader).log v20512 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20512)
>2018-07-11 09:37:10.629170 7f29149e4700  1 mon.magna102@0(leader).log v20512 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 0 entries (version 20512)
>2018-07-11 09:37:11.541172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 66 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:11.712696 7f29149e4700  1 mon.magna102@0(leader).log v20513 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20513)
>2018-07-11 09:37:11.712742 7f29149e4700  1 mon.magna102@0(leader).log v20513 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20513)
>2018-07-11 09:37:16.541436 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 70 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:16.614700 7f29149e4700  1 mon.magna102@0(leader).log v20514 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20514)
>2018-07-11 09:37:16.614745 7f29149e4700  1 mon.magna102@0(leader).log v20514 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20514)
>2018-07-11 09:37:21.541674 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 90 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:22.609093 7f29149e4700  1 mon.magna102@0(leader).log v20515 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20515)
>2018-07-11 09:37:22.609140 7f29149e4700  1 mon.magna102@0(leader).log v20515 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20515)
>2018-07-11 09:37:27.472783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 94 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:27.633776 7f29149e4700  1 mon.magna102@0(leader).log v20516 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20516)
>2018-07-11 09:37:27.633822 7f29149e4700  1 mon.magna102@0(leader).log v20516 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20516)
>2018-07-11 09:37:36.542281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 98 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:37:36.658435 7f29149e4700  1 mon.magna102@0(leader).log v20517 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20517)
>2018-07-11 09:37:36.658480 7f29149e4700  1 mon.magna102@0(leader).log v20517 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20517)
>2018-07-11 09:37:41.542476 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:37:41.659093 7f29149e4700  1 mon.magna102@0(leader).log v20518 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20518)
>2018-07-11 09:37:41.659139 7f29149e4700  1 mon.magna102@0(leader).log v20518 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20518)
>2018-07-11 09:37:43.479684 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 9)
>2018-07-11 09:37:43.634434 7f29149e4700  1 mon.magna102@0(leader).log v20519 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20519)
>2018-07-11 09:37:43.634467 7f29149e4700  1 mon.magna102@0(leader).log v20519 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20519)
>2018-07-11 09:37:46.618523 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10463. Immutable memtables: 0.
>
>2018-07-11 09:37:46.618565 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 891] Syncing log #10460
>2018-07-11 09:37:46.634799 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.618556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:37:46.634811 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 891] Flushing memtable with next log file: 10463
>
>2018-07-11 09:37:46.634837 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866634830, "job": 891, "event": "flush_started", "num_memtables": 1, "num_entries": 762, "num_deletes": 502, "memory_usage": 627624}
>2018-07-11 09:37:46.634840 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 891] Level-0 flush table #10464: started
>2018-07-11 09:37:46.676998 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866676984, "cf_name": "default", "job": 891, "event": "table_file_creation", "file_number": 10464, "file_size": 592530, "table_properties": {"data_size": 586107, "index_size": 2420, "filter_size": 3084, "raw_key_size": 12177, "raw_average_key_size": 20, "raw_value_size": 576452, "raw_average_value_size": 975, "num_data_blocks": 74, "num_entries": 591, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "502", "kMergeOperands": "0"}}
>2018-07-11 09:37:46.677008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 891] Level-0 flush table #10464: 592530 bytes OK
>2018-07-11 09:37:46.727035 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.693531) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10464 started
>2018-07-11 09:37:46.727043 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.726998) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10464: memtable #1 done
>2018-07-11 09:37:46.727046 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.727013) EVENT_LOG_v1 {"time_micros": 1531301866727007, "job": 891, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:37:46.727060 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.727025) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:37:46.727068 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 891] Try to delete WAL files size 608673, prev total WAL file size 608673, number of live WAL files 2.
>
>2018-07-11 09:37:46.727296 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:37:46.727345 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.727338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 09:37:46.727354 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 892] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:37:46.727360 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 891 Base level 0, inputs: [10464(578KB)], [10462(16MB)]
>
>2018-07-11 09:37:46.727374 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866727364, "job": 892, "event": "compaction_started", "files_L0": [10464], "files_L1": [10462], "score": -1, "input_data_size": 17934945}
>2018-07-11 09:37:46.902748 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 892] Generated table #10465: 3146 keys, 15179021 bytes
>2018-07-11 09:37:46.902792 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866902764, "cf_name": "default", "job": 892, "event": "table_file_creation", "file_number": 10465, "file_size": 15179021, "table_properties": {"data_size": 15063280, "index_size": 65970, "filter_size": 48844, "raw_key_size": 63240, "raw_average_key_size": 20, "raw_value_size": 14972357, "raw_average_value_size": 4759, "num_data_blocks": 1937, "num_entries": 3146, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:37:46.944259 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 892] Compacted 1@0 + 1@1 files to L1 => 15179021 bytes
>2018-07-11 09:37:46.977797 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.977730) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.7 rd, 70.0 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.5) out(14.5), read-write-amplify(55.9) write-amplify(25.6) OK, records in: 4158, records dropped: 1012
>
>2018-07-11 09:37:46.977805 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:37:46.977752) EVENT_LOG_v1 {"time_micros": 1531301866977742, "job": 892, "event": "compaction_finished", "compaction_time_micros": 216869, "output_level": 1, "num_output_files": 1, "total_output_size": 15179021, "num_input_records": 4158, "num_output_records": 3146, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:37:46.977999 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866977994, "job": 892, "event": "table_file_deletion", "file_number": 10464}
>2018-07-11 09:37:46.981195 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301866981192, "job": 892, "event": "table_file_deletion", "file_number": 10462}
>2018-07-11 09:37:46.981226 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:37:53.484267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:37:53.662540 7f29149e4700  1 mon.magna102@0(leader).log v20520 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20520)
>2018-07-11 09:37:53.662585 7f29149e4700  1 mon.magna102@0(leader).log v20520 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20520)
>2018-07-11 09:38:01.487998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:01.639495 7f29149e4700  1 mon.magna102@0(leader).log v20521 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20521)
>2018-07-11 09:38:01.639541 7f29149e4700  1 mon.magna102@0(leader).log v20521 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20521)
>2018-07-11 09:38:02.832808 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:38:06.606138 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:06.723267 7f29149e4700  1 mon.magna102@0(leader).log v20522 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20522)
>2018-07-11 09:38:06.723312 7f29149e4700  1 mon.magna102@0(leader).log v20522 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20522)
>2018-07-11 09:38:15.494240 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:15.649176 7f29149e4700  1 mon.magna102@0(leader).log v20523 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20523)
>2018-07-11 09:38:15.649236 7f29149e4700  1 mon.magna102@0(leader).log v20523 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20523)
>2018-07-11 09:38:21.496918 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:21.649854 7f29149e4700  1 mon.magna102@0(leader).log v20524 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20524)
>2018-07-11 09:38:21.649900 7f29149e4700  1 mon.magna102@0(leader).log v20524 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20524)
>2018-07-11 09:38:26.616476 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:26.733618 7f29149e4700  1 mon.magna102@0(leader).log v20525 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20525)
>2018-07-11 09:38:26.733663 7f29149e4700  1 mon.magna102@0(leader).log v20525 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20525)
>2018-07-11 09:38:31.616721 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:32.634397 7f29149e4700  1 mon.magna102@0(leader).log v20526 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20526)
>2018-07-11 09:38:32.634442 7f29149e4700  1 mon.magna102@0(leader).log v20526 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20526)
>2018-07-11 09:38:36.616976 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:36.734897 7f29149e4700  1 mon.magna102@0(leader).log v20527 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20527)
>2018-07-11 09:38:36.734943 7f29149e4700  1 mon.magna102@0(leader).log v20527 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20527)
>2018-07-11 09:38:41.617218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:42.643885 7f29149e4700  1 mon.magna102@0(leader).log v20528 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20528)
>2018-07-11 09:38:42.643930 7f29149e4700  1 mon.magna102@0(leader).log v20528 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20528)
>2018-07-11 09:38:46.617433 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:46.736033 7f29149e4700  1 mon.magna102@0(leader).log v20529 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20529)
>2018-07-11 09:38:46.736078 7f29149e4700  1 mon.magna102@0(leader).log v20529 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20529)
>2018-07-11 09:38:51.617669 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 62 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:52.646228 7f29149e4700  1 mon.magna102@0(leader).log v20530 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20530)
>2018-07-11 09:38:52.646274 7f29149e4700  1 mon.magna102@0(leader).log v20530 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20530)
>2018-07-11 09:38:56.617876 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 66 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:38:56.738373 7f29149e4700  1 mon.magna102@0(leader).log v20531 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20531)
>2018-07-11 09:38:56.738418 7f29149e4700  1 mon.magna102@0(leader).log v20531 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20531)
>2018-07-11 09:39:01.618099 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 90 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 09:39:02.680655 7f29149e4700  1 mon.magna102@0(leader).log v20532 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20532)
>2018-07-11 09:39:02.680701 7f29149e4700  1 mon.magna102@0(leader).log v20532 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20532)
>2018-07-11 09:39:02.832990 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:39:06.618333 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:39:06.747914 7f29149e4700  1 mon.magna102@0(leader).log v20533 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20533)
>2018-07-11 09:39:06.747959 7f29149e4700  1 mon.magna102@0(leader).log v20533 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20533)
>2018-07-11 09:39:07.747912 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 9)
>2018-07-11 09:39:07.923288 7f29149e4700  1 mon.magna102@0(leader).log v20534 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20534)
>2018-07-11 09:39:07.923333 7f29149e4700  1 mon.magna102@0(leader).log v20534 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20534)
>2018-07-11 09:39:21.523311 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 09:39:21.682839 7f29149e4700  1 mon.magna102@0(leader).log v20535 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20535)
>2018-07-11 09:39:21.682883 7f29149e4700  1 mon.magna102@0(leader).log v20535 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20535)
>2018-07-11 09:39:25.525207 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 09:39:25.674947 7f29149e4700  1 mon.magna102@0(leader).log v20536 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20536)
>2018-07-11 09:39:25.674992 7f29149e4700  1 mon.magna102@0(leader).log v20536 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20536)
>2018-07-11 09:39:27.525846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 09:39:27.675306 7f29149e4700  1 mon.magna102@0(leader).log v20537 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20537)
>2018-07-11 09:39:27.675351 7f29149e4700  1 mon.magna102@0(leader).log v20537 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20537)
>2018-07-11 09:39:28.808759 7f29149e4700  1 mon.magna102@0(leader).log v20538 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20538)
>2018-07-11 09:39:28.808792 7f29149e4700  1 mon.magna102@0(leader).log v20538 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 0 entries (version 20538)
>2018-07-11 09:39:33.528383 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 9)
>2018-07-11 09:39:33.676112 7f29149e4700  1 mon.magna102@0(leader).log v20539 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20539)
>2018-07-11 09:39:33.676157 7f29149e4700  1 mon.magna102@0(leader).log v20539 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 1 entries (version 20539)
>2018-07-11 09:39:33.751515 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10466. Immutable memtables: 0.
>
>2018-07-11 09:39:33.751558 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 893] Syncing log #10463
>2018-07-11 09:39:33.768081 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.751549) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:39:33.768092 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 893] Flushing memtable with next log file: 10466
>
>2018-07-11 09:39:33.768119 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301973768111, "job": 893, "event": "flush_started", "num_memtables": 1, "num_entries": 794, "num_deletes": 251, "memory_usage": 1255456}
>2018-07-11 09:39:33.768123 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 893] Level-0 flush table #10467: started
>2018-07-11 09:39:33.810323 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301973810309, "cf_name": "default", "job": 893, "event": "table_file_creation", "file_number": 10467, "file_size": 1202806, "table_properties": {"data_size": 1192561, "index_size": 4962, "filter_size": 4363, "raw_key_size": 8735, "raw_average_key_size": 20, "raw_value_size": 1183103, "raw_average_value_size": 2764, "num_data_blocks": 151, "num_entries": 428, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:39:33.810334 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 893] Level-0 flush table #10467: 1202806 bytes OK
>2018-07-11 09:39:33.860329 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.826851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10467 started
>2018-07-11 09:39:33.860337 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.860293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10467: memtable #1 done
>2018-07-11 09:39:33.860340 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.860307) EVENT_LOG_v1 {"time_micros": 1531301973860301, "job": 893, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:39:33.860341 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.860319) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:39:33.860360 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 893] Try to delete WAL files size 1238206, prev total WAL file size 1238206, number of live WAL files 2.
>
>2018-07-11 09:39:33.860775 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:39:33.860839 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:39:33.860832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:39:33.860846 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 894] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:39:33.860852 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 893 Base level 0, inputs: [10467(1174KB)], [10465(14MB)]
>
>2018-07-11 09:39:33.860865 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301973860855, "job": 894, "event": "compaction_started", "files_L0": [10467], "files_L1": [10465], "score": -1, "input_data_size": 16381827}
>2018-07-11 09:39:34.036025 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 894] Generated table #10468: 3064 keys, 14490100 bytes
>2018-07-11 09:39:34.036070 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301974036043, "cf_name": "default", "job": 894, "event": "table_file_creation", "file_number": 10468, "file_size": 14490100, "table_properties": {"data_size": 14379671, "index_size": 62847, "filter_size": 46655, "raw_key_size": 61728, "raw_average_key_size": 20, "raw_value_size": 14292034, "raw_average_value_size": 4664, "num_data_blocks": 1837, "num_entries": 3064, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:39:34.078779 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 894] Compacted 1@0 + 1@1 files to L1 => 14490100 bytes
>2018-07-11 09:39:34.112290 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:39:34.112224) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.2 rd, 66.5 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.5) out(13.8), read-write-amplify(25.7) write-amplify(12.0) OK, records in: 3574, records dropped: 510
>
>2018-07-11 09:39:34.112299 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:39:34.112245) EVENT_LOG_v1 {"time_micros": 1531301974112236, "job": 894, "event": "compaction_finished", "compaction_time_micros": 217900, "output_level": 1, "num_output_files": 1, "total_output_size": 14490100, "num_input_records": 3574, "num_output_records": 3064, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:39:34.112611 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301974112607, "job": 894, "event": "table_file_deletion", "file_number": 10467}
>2018-07-11 09:39:34.115344 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531301974115341, "job": 894, "event": "table_file_deletion", "file_number": 10465}
>2018-07-11 09:39:40.680033 7f29149e4700  1 mon.magna102@0(leader).log v20540 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20540)
>2018-07-11 09:39:40.680053 7f29149e4700  1 mon.magna102@0(leader).log v20540 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 0 entries (version 20540)
>2018-07-11 09:39:50.664221 7f29149e4700  1 mon.magna102@0(leader).log v20541 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20541)
>2018-07-11 09:39:50.664240 7f29149e4700  1 mon.magna102@0(leader).log v20541 check_sub sending message to client.? 10.8.128.102:0/1750116337 with 0 entries (version 20541)
>2018-07-11 09:40:01.097390 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:01.097413 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3328873957' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:01.215119 7f29149e4700  1 mon.magna102@0(leader).log v20542 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20542)
>2018-07-11 09:40:02.833158 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:40:03.393728 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:03.393752 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1319285347' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:03.507458 7f29149e4700  1 mon.magna102@0(leader).log v20543 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20543)
>2018-07-11 09:40:04.724616 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:04.724639 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2989498024' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:04.841097 7f29149e4700  1 mon.magna102@0(leader).log v20544 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20544)
>2018-07-11 09:40:06.068260 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:06.068284 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1441422543' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:06.974851 7f29149e4700  1 mon.magna102@0(leader).log v20545 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20545)
>2018-07-11 09:40:09.247736 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:09.247760 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2959836124' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:09.366872 7f29149e4700  1 mon.magna102@0(leader).log v20546 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20546)
>2018-07-11 09:40:10.590574 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:10.590597 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1027548864' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:11.500456 7f29149e4700  1 mon.magna102@0(leader).log v20547 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20547)
>2018-07-11 09:40:15.983680 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:15.983703 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1726970419' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:16.684634 7f29149e4700  1 mon.magna102@0(leader).log v20548 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20548)
>2018-07-11 09:40:18.105110 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:18.105133 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1375350634' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:18.818322 7f29149e4700  1 mon.magna102@0(leader).log v20549 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20549)
>2018-07-11 09:40:19.478858 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:40:19.478881 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/229182036' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:40:19.953144 7f29149e4700  1 mon.magna102@0(leader).log v20550 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20550)
>2018-07-11 09:40:38.813254 7f29149e4700  1 mon.magna102@0(leader).log v20551 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20551)
>2018-07-11 09:41:02.833327 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:42:02.833474 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:42:21.811581 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:21.811604 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/19709622' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:22.738558 7f29149e4700  1 mon.magna102@0(leader).log v20552 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20552)
>2018-07-11 09:42:23.441697 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:23.441722 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1243425283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:23.872246 7f29149e4700  1 mon.magna102@0(leader).log v20553 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20553)
>2018-07-11 09:42:25.199281 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:25.199306 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4173495377' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:25.200568 7f29191ed700  1 mon.magna102@0(leader).log v20553 check_sub sending message to client.? 10.8.128.102:0/4173495377 with 1 entries (version 20553)
>2018-07-11 09:42:25.330736 7f29149e4700  1 mon.magna102@0(leader).log v20554 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20554)
>2018-07-11 09:42:25.330756 7f29149e4700  1 mon.magna102@0(leader).log v20554 check_sub sending message to client.? 10.8.128.102:0/4173495377 with 0 entries (version 20554)
>2018-07-11 09:42:29.337844 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:29.337868 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2273397902' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:29.456265 7f29149e4700  1 mon.magna102@0(leader).log v20555 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20555)
>2018-07-11 09:42:30.669062 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:30.669086 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1675113380' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:31.589854 7f29149e4700  1 mon.magna102@0(leader).log v20556 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20556)
>2018-07-11 09:42:31.866222 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:31.866245 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3722955918' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:32.791562 7f29149e4700  1 mon.magna102@0(leader).log v20557 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20557)
>2018-07-11 09:42:33.200147 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:42:33.200170 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1731254242' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:42:33.975218 7f29149e4700  1 mon.magna102@0(leader).log v20558 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20558)
>2018-07-11 09:43:02.833619 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:43:07.502477 7f29149e4700  1 mon.magna102@0(leader).log v20559 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20559)
>2018-07-11 09:43:16.761801 7f29149e4700  1 mon.magna102@0(leader).log v20560 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20560)
>2018-07-11 09:43:37.505055 7f29149e4700  1 mon.magna102@0(leader).log v20561 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20561)
>2018-07-11 09:44:02.113398 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:44:02.113422 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/522704494' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:44:02.783330 7f29149e4700  1 mon.magna102@0(leader).log v20562 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20562)
>2018-07-11 09:44:02.833803 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:44:04.563903 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:44:04.563929 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/793185551' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:44:04.565154 7f29191ed700  1 mon.magna102@0(leader).log v20562 check_sub sending message to client.? 10.8.128.102:0/793185551 with 1 entries (version 20562)
>2018-07-11 09:44:04.933594 7f29149e4700  1 mon.magna102@0(leader).log v20563 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20563)
>2018-07-11 09:44:04.933615 7f29149e4700  1 mon.magna102@0(leader).log v20563 check_sub sending message to client.? 10.8.128.102:0/793185551 with 0 entries (version 20563)
>2018-07-11 09:45:02.833961 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:45:16.781874 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10469. Immutable memtables: 0.
>
>2018-07-11 09:45:16.781918 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 895] Syncing log #10466
>2018-07-11 09:45:16.798365 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.781908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:45:16.798376 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 895] Flushing memtable with next log file: 10469
>
>2018-07-11 09:45:16.798403 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302316798395, "job": 895, "event": "flush_started", "num_memtables": 1, "num_entries": 1644, "num_deletes": 250, "memory_usage": 3052256}
>2018-07-11 09:45:16.798406 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 895] Level-0 flush table #10470: started
>2018-07-11 09:45:16.848897 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302316848883, "cf_name": "default", "job": 895, "event": "table_file_creation", "file_number": 10470, "file_size": 1804964, "table_properties": {"data_size": 1790482, "index_size": 7331, "filter_size": 6231, "raw_key_size": 10745, "raw_average_key_size": 21, "raw_value_size": 1778193, "raw_average_value_size": 3500, "num_data_blocks": 228, "num_entries": 508, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:45:16.848907 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 895] Level-0 flush table #10470: 1804964 bytes OK
>2018-07-11 09:45:16.898932 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.865426) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10470 started
>2018-07-11 09:45:16.898940 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.898896) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10470: memtable #1 done
>2018-07-11 09:45:16.898943 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.898910) EVENT_LOG_v1 {"time_micros": 1531302316898904, "job": 895, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:45:16.898957 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.898922) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:45:16.898964 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 895] Try to delete WAL files size 3020185, prev total WAL file size 3020185, number of live WAL files 2.
>
>2018-07-11 09:45:16.899736 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:45:16.899797 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:45:16.899790) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:45:16.899804 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 896] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:45:16.899811 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 895 Base level 0, inputs: [10470(1762KB)], [10468(13MB)]
>
>2018-07-11 09:45:16.899824 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302316899814, "job": 896, "event": "compaction_started", "files_L0": [10470], "files_L1": [10468], "score": -1, "input_data_size": 16295064}
>2018-07-11 09:45:17.091309 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 896] Generated table #10471: 3229 keys, 15707131 bytes
>2018-07-11 09:45:17.091354 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302317091338, "cf_name": "default", "job": 896, "event": "table_file_creation", "file_number": 10471, "file_size": 15707131, "table_properties": {"data_size": 15588431, "index_size": 67428, "filter_size": 50345, "raw_key_size": 64894, "raw_average_key_size": 20, "raw_value_size": 15495318, "raw_average_value_size": 4798, "num_data_blocks": 1979, "num_entries": 3229, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:45:17.132827 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 896] Compacted 1@0 + 1@1 files to L1 => 15707131 bytes
>2018-07-11 09:45:17.166352 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:45:17.166285) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.9 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.8) out(15.0), read-write-amplify(17.7) write-amplify(8.7) OK, records in: 3572, records dropped: 343
>
>2018-07-11 09:45:17.166361 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:45:17.166307) EVENT_LOG_v1 {"time_micros": 1531302317166297, "job": 896, "event": "compaction_finished", "compaction_time_micros": 232986, "output_level": 1, "num_output_files": 1, "total_output_size": 15707131, "num_input_records": 3572, "num_output_records": 3229, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:45:17.166798 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302317166794, "job": 896, "event": "table_file_deletion", "file_number": 10470}
>2018-07-11 09:45:17.169494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302317169491, "job": 896, "event": "table_file_deletion", "file_number": 10468}
>2018-07-11 09:45:17.169609 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:45:54.549657 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:45:54.549681 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/349172466' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:45:54.829339 7f29149e4700  1 mon.magna102@0(leader).log v20564 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20564)
>2018-07-11 09:46:02.834116 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:46:11.055529 7f29149e4700  1 mon.magna102@0(leader).log v20565 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20565)
>2018-07-11 09:46:47.948263 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:46:47.948289 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1708369783' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:46:47.949550 7f29191ed700  1 mon.magna102@0(leader).log v20565 check_sub sending message to client.? 10.8.128.102:0/1708369783 with 1 entries (version 20565)
>2018-07-11 09:46:48.866779 7f29149e4700  1 mon.magna102@0(leader).log v20566 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20566)
>2018-07-11 09:46:48.866799 7f29149e4700  1 mon.magna102@0(leader).log v20566 check_sub sending message to client.? 10.8.128.102:0/1708369783 with 0 entries (version 20566)
>2018-07-11 09:46:51.018288 7f29149e4700  1 mon.magna102@0(leader).log v20567 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20567)
>2018-07-11 09:46:51.018308 7f29149e4700  1 mon.magna102@0(leader).log v20567 check_sub sending message to client.? 10.8.128.102:0/1708369783 with 0 entries (version 20567)
>2018-07-11 09:46:55.885839 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10472. Immutable memtables: 0.
>
>2018-07-11 09:46:55.885885 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 897] Syncing log #10469
>2018-07-11 09:46:55.902405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.885876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:46:55.902417 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 897] Flushing memtable with next log file: 10472
>
>2018-07-11 09:46:55.902430 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302415902423, "job": 897, "event": "flush_started", "num_memtables": 1, "num_entries": 638, "num_deletes": 251, "memory_usage": 845488}
>2018-07-11 09:46:55.902433 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 897] Level-0 flush table #10473: started
>2018-07-11 09:46:55.944604 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302415944590, "cf_name": "default", "job": 897, "event": "table_file_creation", "file_number": 10473, "file_size": 806393, "table_properties": {"data_size": 798682, "index_size": 3603, "filter_size": 3189, "raw_key_size": 7591, "raw_average_key_size": 20, "raw_value_size": 791000, "raw_average_value_size": 2126, "num_data_blocks": 110, "num_entries": 372, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:46:55.944614 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 897] Level-0 flush table #10473: 806393 bytes OK
>2018-07-11 09:46:55.994617 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.961168) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10473 started
>2018-07-11 09:46:55.994625 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.994581) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10473: memtable #1 done
>2018-07-11 09:46:55.994627 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.994595) EVENT_LOG_v1 {"time_micros": 1531302415994589, "job": 897, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:46:55.994629 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.994607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:46:55.994636 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 897] Try to delete WAL files size 831139, prev total WAL file size 831139, number of live WAL files 2.
>
>2018-07-11 09:46:55.994892 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:46:55.994946 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:46:55.994939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:46:55.994955 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 898] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:46:55.994961 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 897 Base level 0, inputs: [10473(787KB)], [10471(14MB)]
>
>2018-07-11 09:46:55.994973 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302415994964, "job": 898, "event": "compaction_started", "files_L0": [10473], "files_L1": [10471], "score": -1, "input_data_size": 16513524}
>2018-07-11 09:46:56.170361 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 898] Generated table #10474: 3091 keys, 14518145 bytes
>2018-07-11 09:46:56.170397 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302416170382, "cf_name": "default", "job": 898, "event": "table_file_creation", "file_number": 10474, "file_size": 14518145, "table_properties": {"data_size": 14407576, "index_size": 62873, "filter_size": 46769, "raw_key_size": 62238, "raw_average_key_size": 20, "raw_value_size": 14319544, "raw_average_value_size": 4632, "num_data_blocks": 1837, "num_entries": 3091, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:46:56.211893 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 898] Compacted 1@0 + 1@1 files to L1 => 14518145 bytes
>2018-07-11 09:46:56.245429 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:46:56.245355) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.1 rd, 66.9 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.0) out(13.8), read-write-amplify(38.5) write-amplify(18.0) OK, records in: 3601, records dropped: 510
>
>2018-07-11 09:46:56.245437 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:46:56.245377) EVENT_LOG_v1 {"time_micros": 1531302416245368, "job": 898, "event": "compaction_finished", "compaction_time_micros": 216903, "output_level": 1, "num_output_files": 1, "total_output_size": 14518145, "num_input_records": 3601, "num_output_records": 3091, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:46:56.245653 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302416245649, "job": 898, "event": "table_file_deletion", "file_number": 10473}
>2018-07-11 09:46:56.248664 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302416248661, "job": 898, "event": "table_file_deletion", "file_number": 10471}
>2018-07-11 09:47:02.834300 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:47:07.858829 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:47:07.858855 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4178154985' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:47:07.860232 7f29191ed700  1 mon.magna102@0(leader).log v20567 check_sub sending message to client.? 10.8.128.102:0/4178154985 with 1 entries (version 20567)
>2018-07-11 09:47:08.896811 7f29149e4700  1 mon.magna102@0(leader).log v20568 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20568)
>2018-07-11 09:47:08.896844 7f29149e4700  1 mon.magna102@0(leader).log v20568 check_sub sending message to client.? 10.8.128.102:0/4178154985 with 0 entries (version 20568)
>2018-07-11 09:47:11.055438 7f29149e4700  1 mon.magna102@0(leader).log v20569 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20569)
>2018-07-11 09:47:11.055471 7f29149e4700  1 mon.magna102@0(leader).log v20569 check_sub sending message to client.? 10.8.128.102:0/4178154985 with 0 entries (version 20569)
>2018-07-11 09:47:40.477299 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:47:40.477325 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/96138838' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:47:40.478645 7f29191ed700  1 mon.magna102@0(leader).log v20569 check_sub sending message to client.? 10.8.128.102:0/96138838 with 1 entries (version 20569)
>2018-07-11 09:47:40.875748 7f29149e4700  1 mon.magna102@0(leader).log v20570 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20570)
>2018-07-11 09:47:40.875781 7f29149e4700  1 mon.magna102@0(leader).log v20570 check_sub sending message to client.? 10.8.128.102:0/96138838 with 0 entries (version 20570)
>2018-07-11 09:47:51.051555 7f29149e4700  1 mon.magna102@0(leader).log v20571 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20571)
>2018-07-11 09:48:01.060885 7f29149e4700  1 mon.magna102@0(leader).log v20572 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20572)
>2018-07-11 09:48:02.834480 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:48:25.065005 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:48:25.065028 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3542550699' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:48:25.187816 7f29149e4700  1 mon.magna102@0(leader).log v20573 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20573)
>2018-07-11 09:48:31.063365 7f29149e4700  1 mon.magna102@0(leader).log v20574 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20574)
>2018-07-11 09:48:32.651432 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 09:48:32.651464 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/393485106' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 09:48:32.652857 7f29191ed700  1 mon.magna102@0(leader).log v20574 check_sub sending message to client.? 10.8.128.102:0/393485106 with 1 entries (version 20574)
>2018-07-11 09:48:33.230430 7f29149e4700  1 mon.magna102@0(leader).log v20575 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20575)
>2018-07-11 09:48:33.230451 7f29149e4700  1 mon.magna102@0(leader).log v20575 check_sub sending message to client.? 10.8.128.102:0/393485106 with 0 entries (version 20575)
>2018-07-11 09:49:02.834650 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:50:02.834832 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:51:02.835034 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:52:02.835216 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:53:02.835398 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:53:37.008523 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10475. Immutable memtables: 0.
>
>2018-07-11 09:53:37.008565 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 899] Syncing log #10472
>2018-07-11 09:53:37.024979 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.008556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:53:37.024991 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 899] Flushing memtable with next log file: 10475
>
>2018-07-11 09:53:37.025004 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817024997, "job": 899, "event": "flush_started", "num_memtables": 1, "num_entries": 1719, "num_deletes": 250, "memory_usage": 3172904}
>2018-07-11 09:53:37.025007 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 899] Level-0 flush table #10476: started
>2018-07-11 09:53:37.083737 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817083722, "cf_name": "default", "job": 899, "event": "table_file_creation", "file_number": 10476, "file_size": 1711416, "table_properties": {"data_size": 1697358, "index_size": 7159, "filter_size": 5979, "raw_key_size": 10349, "raw_average_key_size": 21, "raw_value_size": 1685470, "raw_average_value_size": 3446, "num_data_blocks": 224, "num_entries": 489, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 09:53:37.083748 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 899] Level-0 flush table #10476: 1711416 bytes OK
>2018-07-11 09:53:37.133841 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.100413) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10476 started
>2018-07-11 09:53:37.133851 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.133804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10476: memtable #1 done
>2018-07-11 09:53:37.133856 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.133819) EVENT_LOG_v1 {"time_micros": 1531302817133813, "job": 899, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:53:37.133858 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.133831) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:53:37.133865 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 899] Try to delete WAL files size 3140225, prev total WAL file size 3140225, number of live WAL files 2.
>
>2018-07-11 09:53:37.134676 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:53:37.134855 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 09:53:37.134861 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 64844.0 total, 1000.4 interval
>Cumulative writes: 88K writes, 417K keys, 88K commit groups, 1.0 writes per commit group, ingest: 0.69 GB, 0.01 MB/s
>Cumulative WAL: 88K writes, 88K syncs, 1.00 writes per sync, written: 0.69 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1142 writes, 5557 keys, 1142 commit groups, 1.0 writes per commit group, ingest: 8.43 MB, 0.01 MB/s
>Interval WAL: 1142 writes, 1142 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.63 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       450    0.081       0      0
>  L1      1/1   13.85 MB   0.0      7.1     0.6      6.5       6.5     -0.0       0.0  10.7     72.6     66.4       100       448    0.223   1599K   191K
> Sum      2/2   15.48 MB   0.0      7.1     0.6      6.5       7.1      0.6       0.0  11.7     53.1     53.1       136       898    0.152   1599K   191K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.4     56.2     55.0         2        12    0.145     22K   2791
>Uptime(secs): 64844.0 total, 64844.0 interval
>Flush(GB): cumulative 0.604, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.08 GB write, 0.11 MB/s write, 7.08 GB read, 0.11 MB/s read, 136.3 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.10 GB read, 0.00 MB/s read, 1.7 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.63 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        36       450    0.081       0      0
>  L1      1/1   13.85 MB   0.0      7.1     0.6      6.5       6.5     -0.0       0.0  10.7     72.6     66.4       100       448    0.223   1599K   191K
> Sum      2/2   15.48 MB   0.0      7.1     0.6      6.5       7.1      0.6       0.0  11.7     53.1     53.1       136       898    0.152   1599K   191K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 64844.0 total, 64844.0 interval
>Flush(GB): cumulative 0.604, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.08 GB write, 0.11 MB/s write, 7.08 GB read, 0.11 MB/s read, 136.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 09:53:37.134882 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.134875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 09:53:37.134886 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 900] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:53:37.134892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 899 Base level 0, inputs: [10476(1671KB)], [10474(13MB)]
>
>2018-07-11 09:53:37.134904 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817134895, "job": 900, "event": "compaction_started", "files_L0": [10476], "files_L1": [10474], "score": -1, "input_data_size": 16229561}
>2018-07-11 09:53:37.326316 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 900] Generated table #10477: 3266 keys, 15842160 bytes
>2018-07-11 09:53:37.326354 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817326339, "cf_name": "default", "job": 900, "event": "table_file_creation", "file_number": 10477, "file_size": 15842160, "table_properties": {"data_size": 15722053, "index_size": 68319, "filter_size": 50861, "raw_key_size": 65646, "raw_average_key_size": 20, "raw_value_size": 15627787, "raw_average_value_size": 4784, "num_data_blocks": 2005, "num_entries": 3266, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:53:37.367834 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 900] Compacted 1@0 + 1@1 files to L1 => 15842160 bytes
>2018-07-11 09:53:37.401348 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.401275) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.7 rd, 68.0 wr, level 1, files in(1, 1) out(1) MB in(1.6, 13.8) out(15.1), read-write-amplify(18.7) write-amplify(9.3) OK, records in: 3580, records dropped: 314
>
>2018-07-11 09:53:37.401357 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:53:37.401296) EVENT_LOG_v1 {"time_micros": 1531302817401287, "job": 900, "event": "compaction_finished", "compaction_time_micros": 232914, "output_level": 1, "num_output_files": 1, "total_output_size": 15842160, "num_input_records": 3580, "num_output_records": 3266, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:53:37.401753 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817401749, "job": 900, "event": "table_file_deletion", "file_number": 10476}
>2018-07-11 09:53:37.404307 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302817404304, "job": 900, "event": "table_file_deletion", "file_number": 10474}
>2018-07-11 09:53:37.404360 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:54:02.835585 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 09:54:17.046353 7f29149e4700  1 mon.magna102@0(leader).log v20576 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20576)
>2018-07-11 09:54:17.046374 7f29149e4700  1 mon.magna102@0(leader).log v20576 check_sub sending message to client.? 10.8.128.102:0/393485106 with 0 entries (version 20576)
>2018-07-11 09:54:47.491435 7f29149e4700  1 mon.magna102@0(leader).log v20577 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20577)
>2018-07-11 09:54:47.491454 7f29149e4700  1 mon.magna102@0(leader).log v20577 check_sub sending message to client.? 10.8.128.102:0/393485106 with 0 entries (version 20577)
>2018-07-11 09:54:54.067564 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10478. Immutable memtables: 0.
>
>2018-07-11 09:54:54.067608 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 901] Syncing log #10475
>2018-07-11 09:54:54.084131 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.067598) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 09:54:54.084143 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 901] Flushing memtable with next log file: 10478
>
>2018-07-11 09:54:54.084157 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894084149, "job": 901, "event": "flush_started", "num_memtables": 1, "num_entries": 545, "num_deletes": 251, "memory_usage": 642520}
>2018-07-11 09:54:54.084160 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 901] Level-0 flush table #10479: started
>2018-07-11 09:54:54.126343 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894126329, "cf_name": "default", "job": 901, "event": "table_file_creation", "file_number": 10479, "file_size": 611124, "table_properties": {"data_size": 604888, "index_size": 2743, "filter_size": 2574, "raw_key_size": 7007, "raw_average_key_size": 20, "raw_value_size": 598227, "raw_average_value_size": 1739, "num_data_blocks": 84, "num_entries": 344, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 09:54:54.126353 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 901] Level-0 flush table #10479: 611124 bytes OK
>2018-07-11 09:54:54.184679 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.142884) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10479 started
>2018-07-11 09:54:54.184687 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.184643) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10479: memtable #1 done
>2018-07-11 09:54:54.184689 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.184658) EVENT_LOG_v1 {"time_micros": 1531302894184652, "job": 901, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 09:54:54.184694 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.184669) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 09:54:54.184701 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 901] Try to delete WAL files size 629972, prev total WAL file size 629972, number of live WAL files 2.
>
>2018-07-11 09:54:54.184900 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 09:54:54.184952 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.184945) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 09:54:54.184962 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 902] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 09:54:54.184968 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 901 Base level 0, inputs: [10479(596KB)], [10477(15MB)]
>
>2018-07-11 09:54:54.184980 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894184971, "job": 902, "event": "compaction_started", "files_L0": [10479], "files_L1": [10477], "score": -1, "input_data_size": 16453284}
>2018-07-11 09:54:54.360401 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 902] Generated table #10480: 3100 keys, 14342936 bytes
>2018-07-11 09:54:54.360432 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894360418, "cf_name": "default", "job": 902, "event": "table_file_creation", "file_number": 10480, "file_size": 14342936, "table_properties": {"data_size": 14232687, "index_size": 62876, "filter_size": 46446, "raw_key_size": 62406, "raw_average_key_size": 20, "raw_value_size": 14144530, "raw_average_value_size": 4562, "num_data_blocks": 1837, "num_entries": 3100, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 09:54:54.401919 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 902] Compacted 1@0 + 1@1 files to L1 => 14342936 bytes
>2018-07-11 09:54:54.443786 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.443718) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.8 rd, 66.1 wr, level 1, files in(1, 1) out(1) MB in(0.6, 15.1) out(13.7), read-write-amplify(50.4) write-amplify(23.5) OK, records in: 3610, records dropped: 510
>
>2018-07-11 09:54:54.443795 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-09:54:54.443740) EVENT_LOG_v1 {"time_micros": 1531302894443730, "job": 902, "event": "compaction_finished", "compaction_time_micros": 216923, "output_level": 1, "num_output_files": 1, "total_output_size": 14342936, "num_input_records": 3610, "num_output_records": 3100, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 09:54:54.443962 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894443958, "job": 902, "event": "table_file_deletion", "file_number": 10479}
>2018-07-11 09:54:54.446784 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531302894446780, "job": 902, "event": "table_file_deletion", "file_number": 10477}
>2018-07-11 09:54:55.236063 7f29149e4700  1 mon.magna102@0(leader).log v20578 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20578)
>2018-07-11 09:54:55.236083 7f29149e4700  1 mon.magna102@0(leader).log v20578 check_sub sending message to client.? 10.8.128.102:0/393485106 with 0 entries (version 20578)
>2018-07-11 09:55:02.835770 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:56:02.835941 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:57:02.836109 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:57:57.494109 7f29149e4700  1 mon.magna102@0(leader).log v20579 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20579)
>2018-07-11 09:57:57.494131 7f29149e4700  1 mon.magna102@0(leader).log v20579 check_sub sending message to client.? 10.8.128.102:0/393485106 with 0 entries (version 20579)
>2018-07-11 09:58:02.836257 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 09:59:02.836430 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:00:02.836608 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:00:20.374387 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:00:20.374415 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1642608779' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:00:20.375699 7f29191ed700  1 mon.magna102@0(leader).log v20579 check_sub sending message to client.? 10.8.128.102:0/1642608779 with 1 entries (version 20579)
>2018-07-11 10:00:21.215328 7f29149e4700  1 mon.magna102@0(leader).log v20580 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20580)
>2018-07-11 10:00:21.215348 7f29149e4700  1 mon.magna102@0(leader).log v20580 check_sub sending message to client.? 10.8.128.102:0/1642608779 with 0 entries (version 20580)
>2018-07-11 10:00:31.216132 7f29149e4700  1 mon.magna102@0(leader).log v20581 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20581)
>2018-07-11 10:01:02.836775 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:01:16.245056 7f29149e4700  1 mon.magna102@0(leader).log v20582 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20582)
>2018-07-11 10:01:18.839820 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:01:18.839845 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3575177126' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:01:18.841225 7f29191ed700  1 mon.magna102@0(leader).log v20582 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20582)
>2018-07-11 10:01:19.238296 7f29149e4700  1 mon.magna102@0(leader).log v20583 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20583)
>2018-07-11 10:01:19.238316 7f29149e4700  1 mon.magna102@0(leader).log v20583 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 0 entries (version 20583)
>2018-07-11 10:01:57.225043 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10481. Immutable memtables: 0.
>
>2018-07-11 10:01:57.225137 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 903] Syncing log #10478
>2018-07-11 10:01:57.241262 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.225127) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:01:57.241273 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 903] Flushing memtable with next log file: 10481
>
>2018-07-11 10:01:57.241300 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317241292, "job": 903, "event": "flush_started", "num_memtables": 1, "num_entries": 1780, "num_deletes": 250, "memory_usage": 3294392}
>2018-07-11 10:01:57.241303 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 903] Level-0 flush table #10482: started
>2018-07-11 10:01:57.300129 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317300115, "cf_name": "default", "job": 903, "event": "table_file_creation", "file_number": 10482, "file_size": 1750699, "table_properties": {"data_size": 1736291, "index_size": 7375, "filter_size": 6113, "raw_key_size": 10447, "raw_average_key_size": 21, "raw_value_size": 1724175, "raw_average_value_size": 3490, "num_data_blocks": 231, "num_entries": 494, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:01:57.300140 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 903] Level-0 flush table #10482: 1750699 bytes OK
>2018-07-11 10:01:57.358536 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.316653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10482 started
>2018-07-11 10:01:57.358545 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.358463) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10482: memtable #1 done
>2018-07-11 10:01:57.358548 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.358478) EVENT_LOG_v1 {"time_micros": 1531303317358472, "job": 903, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:01:57.358550 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.358490) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:01:57.358571 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 903] Try to delete WAL files size 3260067, prev total WAL file size 3260067, number of live WAL files 2.
>
>2018-07-11 10:01:57.359352 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:01:57.359402 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.359395) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:01:57.359412 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 904] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:01:57.359419 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 903 Base level 0, inputs: [10482(1709KB)], [10480(13MB)]
>
>2018-07-11 10:01:57.359432 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317359422, "job": 904, "event": "compaction_started", "files_L0": [10482], "files_L1": [10480], "score": -1, "input_data_size": 16093635}
>2018-07-11 10:01:57.542560 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 904] Generated table #10483: 3291 keys, 15781405 bytes
>2018-07-11 10:01:57.542610 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317542583, "cf_name": "default", "job": 904, "event": "table_file_creation", "file_number": 10483, "file_size": 15781405, "table_properties": {"data_size": 15660664, "index_size": 68900, "filter_size": 50914, "raw_key_size": 66154, "raw_average_key_size": 20, "raw_value_size": 15565644, "raw_average_value_size": 4729, "num_data_blocks": 2022, "num_entries": 3291, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:01:57.575727 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 904] Compacted 1@0 + 1@1 files to L1 => 15781405 bytes
>2018-07-11 10:01:57.617587 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.617518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.4 rd, 73.0 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.7) out(15.1), read-write-amplify(18.2) write-amplify(9.0) OK, records in: 3594, records dropped: 303
>
>2018-07-11 10:01:57.617596 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:01:57.617540) EVENT_LOG_v1 {"time_micros": 1531303317617530, "job": 904, "event": "compaction_finished", "compaction_time_micros": 216279, "output_level": 1, "num_output_files": 1, "total_output_size": 15781405, "num_input_records": 3594, "num_output_records": 3291, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:01:57.618002 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317617998, "job": 904, "event": "table_file_deletion", "file_number": 10482}
>2018-07-11 10:01:57.620545 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303317620542, "job": 904, "event": "table_file_deletion", "file_number": 10480}
>2018-07-11 10:01:57.620664 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:02:02.836966 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:03:00.290828 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10484. Immutable memtables: 0.
>
>2018-07-11 10:03:00.290871 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 905] Syncing log #10481
>2018-07-11 10:03:00.307405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.290861) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:03:00.307416 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 905] Flushing memtable with next log file: 10484
>
>2018-07-11 10:03:00.307442 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380307434, "job": 905, "event": "flush_started", "num_memtables": 1, "num_entries": 480, "num_deletes": 251, "memory_usage": 498928}
>2018-07-11 10:03:00.307445 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 905] Level-0 flush table #10485: started
>2018-07-11 10:03:00.349623 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380349610, "cf_name": "default", "job": 905, "event": "table_file_creation", "file_number": 10485, "file_size": 472463, "table_properties": {"data_size": 467248, "index_size": 2154, "filter_size": 2142, "raw_key_size": 6540, "raw_average_key_size": 20, "raw_value_size": 461350, "raw_average_value_size": 1432, "num_data_blocks": 66, "num_entries": 322, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:03:00.349633 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 905] Level-0 flush table #10485: 472463 bytes OK
>2018-07-11 10:03:00.399645 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.366130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10485 started
>2018-07-11 10:03:00.399654 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.399610) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10485: memtable #1 done
>2018-07-11 10:03:00.399656 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.399625) EVENT_LOG_v1 {"time_micros": 1531303380399618, "job": 905, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:03:00.399658 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.399636) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:03:00.399677 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 905] Try to delete WAL files size 487617, prev total WAL file size 487617, number of live WAL files 2.
>
>2018-07-11 10:03:00.399863 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:03:00.399913 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.399905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:03:00.399922 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 906] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:03:00.399928 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 905 Base level 0, inputs: [10485(461KB)], [10483(15MB)]
>
>2018-07-11 10:03:00.399942 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380399932, "job": 906, "event": "compaction_started", "files_L0": [10485], "files_L1": [10483], "score": -1, "input_data_size": 16253868}
>2018-07-11 10:03:00.575364 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 906] Generated table #10486: 3105 keys, 14299192 bytes
>2018-07-11 10:03:00.575397 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380575382, "cf_name": "default", "job": 906, "event": "table_file_creation", "file_number": 10486, "file_size": 14299192, "table_properties": {"data_size": 14189050, "index_size": 62847, "filter_size": 46368, "raw_key_size": 62498, "raw_average_key_size": 20, "raw_value_size": 14100848, "raw_average_value_size": 4541, "num_data_blocks": 1836, "num_entries": 3105, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:03:00.616846 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 906] Compacted 1@0 + 1@1 files to L1 => 14299192 bytes
>2018-07-11 10:03:00.650352 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.650285) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.9 rd, 65.9 wr, level 1, files in(1, 1) out(1) MB in(0.5, 15.1) out(13.6), read-write-amplify(64.7) write-amplify(30.3) OK, records in: 3613, records dropped: 508
>
>2018-07-11 10:03:00.650360 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:03:00.650307) EVENT_LOG_v1 {"time_micros": 1531303380650297, "job": 906, "event": "compaction_finished", "compaction_time_micros": 216887, "output_level": 1, "num_output_files": 1, "total_output_size": 14299192, "num_input_records": 3613, "num_output_records": 3105, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:03:00.650526 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380650522, "job": 906, "event": "table_file_deletion", "file_number": 10485}
>2018-07-11 10:03:00.653429 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303380653426, "job": 906, "event": "table_file_deletion", "file_number": 10483}
>2018-07-11 10:03:02.837139 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:04:02.837314 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:05:02.837477 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:06:02.837643 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:06:05.358845 7f29149e4700  1 mon.magna102@0(leader).log v20584 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20584)
>2018-07-11 10:06:05.358867 7f29149e4700  1 mon.magna102@0(leader).log v20584 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 0 entries (version 20584)
>2018-07-11 10:06:20.233797 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 10:06:20.385342 7f29149e4700  1 mon.magna102@0(leader).log v20585 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20585)
>2018-07-11 10:06:20.385386 7f29149e4700  1 mon.magna102@0(leader).log v20585 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20585)
>2018-07-11 10:06:26.236415 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:26.394111 7f29149e4700  1 mon.magna102@0(leader).log v20586 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20586)
>2018-07-11 10:06:26.394156 7f29149e4700  1 mon.magna102@0(leader).log v20586 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20586)
>2018-07-11 10:06:32.239044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:32.411370 7f29149e4700  1 mon.magna102@0(leader).log v20587 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20587)
>2018-07-11 10:06:32.411415 7f29149e4700  1 mon.magna102@0(leader).log v20587 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20587)
>2018-07-11 10:06:37.277933 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:37.395138 7f29149e4700  1 mon.magna102@0(leader).log v20588 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20588)
>2018-07-11 10:06:37.395196 7f29149e4700  1 mon.magna102@0(leader).log v20588 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20588)
>2018-07-11 10:06:46.245282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:46.397292 7f29149e4700  1 mon.magna102@0(leader).log v20589 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20589)
>2018-07-11 10:06:46.397337 7f29149e4700  1 mon.magna102@0(leader).log v20589 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20589)
>2018-07-11 10:06:52.247863 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:52.406143 7f29149e4700  1 mon.magna102@0(leader).log v20590 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20590)
>2018-07-11 10:06:52.406177 7f29149e4700  1 mon.magna102@0(leader).log v20590 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20590)
>2018-07-11 10:06:57.289256 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:06:57.381675 7f29149e4700  1 mon.magna102@0(leader).log v20591 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20591)
>2018-07-11 10:06:57.381720 7f29149e4700  1 mon.magna102@0(leader).log v20591 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20591)
>2018-07-11 10:07:02.837803 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:07:07.307211 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:07:07.390973 7f29149e4700  1 mon.magna102@0(leader).log v20592 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20592)
>2018-07-11 10:07:07.391018 7f29149e4700  1 mon.magna102@0(leader).log v20592 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20592)
>2018-07-11 10:07:32.265578 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 5,10 (REQUEST_SLOW)
>2018-07-11 10:07:32.418144 7f29149e4700  1 mon.magna102@0(leader).log v20593 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20593)
>2018-07-11 10:07:32.418176 7f29149e4700  1 mon.magna102@0(leader).log v20593 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20593)
>2018-07-11 10:07:50.273647 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:07:50.429104 7f29149e4700  1 mon.magna102@0(leader).log v20594 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20594)
>2018-07-11 10:07:50.429149 7f29149e4700  1 mon.magna102@0(leader).log v20594 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20594)
>2018-07-11 10:07:57.312591 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:07:57.413014 7f29149e4700  1 mon.magna102@0(leader).log v20595 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20595)
>2018-07-11 10:07:57.413060 7f29149e4700  1 mon.magna102@0(leader).log v20595 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20595)
>2018-07-11 10:08:02.837972 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:08:07.313508 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:07.430585 7f29149e4700  1 mon.magna102@0(leader).log v20596 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20596)
>2018-07-11 10:08:07.430629 7f29149e4700  1 mon.magna102@0(leader).log v20596 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20596)
>2018-07-11 10:08:32.292384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:32.466212 7f29149e4700  1 mon.magna102@0(leader).log v20597 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20597)
>2018-07-11 10:08:32.466246 7f29149e4700  1 mon.magna102@0(leader).log v20597 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20597)
>2018-07-11 10:08:37.332715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 40 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:37.433297 7f29149e4700  1 mon.magna102@0(leader).log v20598 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20598)
>2018-07-11 10:08:37.433342 7f29149e4700  1 mon.magna102@0(leader).log v20598 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20598)
>2018-07-11 10:08:46.298674 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 44 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:46.468825 7f29149e4700  1 mon.magna102@0(leader).log v20599 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20599)
>2018-07-11 10:08:46.468870 7f29149e4700  1 mon.magna102@0(leader).log v20599 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20599)
>2018-07-11 10:08:52.301376 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 46 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:52.444542 7f29149e4700  1 mon.magna102@0(leader).log v20600 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20600)
>2018-07-11 10:08:52.444586 7f29149e4700  1 mon.magna102@0(leader).log v20600 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20600)
>2018-07-11 10:08:57.334200 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:08:57.436652 7f29149e4700  1 mon.magna102@0(leader).log v20601 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20601)
>2018-07-11 10:08:57.436696 7f29149e4700  1 mon.magna102@0(leader).log v20601 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20601)
>2018-07-11 10:09:02.838133 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:09:06.307610 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:09:06.462534 7f29149e4700  1 mon.magna102@0(leader).log v20602 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20602)
>2018-07-11 10:09:06.462577 7f29149e4700  1 mon.magna102@0(leader).log v20602 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20602)
>2018-07-11 10:09:12.310238 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 60 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:09:12.463115 7f29149e4700  1 mon.magna102@0(leader).log v20603 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20603)
>2018-07-11 10:09:12.463161 7f29149e4700  1 mon.magna102@0(leader).log v20603 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20603)
>2018-07-11 10:09:17.346325 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 64 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:09:17.446932 7f29149e4700  1 mon.magna102@0(leader).log v20604 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20604)
>2018-07-11 10:09:17.446976 7f29149e4700  1 mon.magna102@0(leader).log v20604 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20604)
>2018-07-11 10:09:26.316531 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 70 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:09:26.472845 7f29149e4700  1 mon.magna102@0(leader).log v20605 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20605)
>2018-07-11 10:09:26.472888 7f29149e4700  1 mon.magna102@0(leader).log v20605 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20605)
>2018-07-11 10:09:32.319173 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 70 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:09:32.473426 7f29149e4700  1 mon.magna102@0(leader).log v20606 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20606)
>2018-07-11 10:09:32.473470 7f29149e4700  1 mon.magna102@0(leader).log v20606 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20606)
>2018-07-11 10:09:46.325448 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:09:46.492447 7f29149e4700  1 mon.magna102@0(leader).log v20607 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20607)
>2018-07-11 10:09:46.492491 7f29149e4700  1 mon.magna102@0(leader).log v20607 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20607)
>2018-07-11 10:09:52.367715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:09:52.501404 7f29149e4700  1 mon.magna102@0(leader).log v20608 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20608)
>2018-07-11 10:09:52.501449 7f29149e4700  1 mon.magna102@0(leader).log v20608 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20608)
>2018-07-11 10:09:57.367936 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 12 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:09:57.485199 7f29149e4700  1 mon.magna102@0(leader).log v20609 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20609)
>2018-07-11 10:09:57.485245 7f29149e4700  1 mon.magna102@0(leader).log v20609 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20609)
>2018-07-11 10:10:02.838310 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:10:06.334495 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:10:06.511337 7f29149e4700  1 mon.magna102@0(leader).log v20610 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20610)
>2018-07-11 10:10:06.511383 7f29149e4700  1 mon.magna102@0(leader).log v20610 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20610)
>2018-07-11 10:10:12.337125 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 28 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:10:12.486912 7f29149e4700  1 mon.magna102@0(leader).log v20611 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20611)
>2018-07-11 10:10:12.486956 7f29149e4700  1 mon.magna102@0(leader).log v20611 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20611)
>2018-07-11 10:10:17.377950 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:10:17.446027 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10487. Immutable memtables: 0.
>
>2018-07-11 10:10:17.446071 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 907] Syncing log #10484
>2018-07-11 10:10:17.462455 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.446061) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:10:17.462466 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 907] Flushing memtable with next log file: 10487
>
>2018-07-11 10:10:17.462493 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817462484, "job": 907, "event": "flush_started", "num_memtables": 1, "num_entries": 2005, "num_deletes": 250, "memory_usage": 3877976}
>2018-07-11 10:10:17.462496 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 907] Level-0 flush table #10488: started
>2018-07-11 10:10:17.546549 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817546534, "cf_name": "default", "job": 907, "event": "table_file_creation", "file_number": 10488, "file_size": 2249544, "table_properties": {"data_size": 2231911, "index_size": 9113, "filter_size": 7596, "raw_key_size": 11929, "raw_average_key_size": 21, "raw_value_size": 2217532, "raw_average_value_size": 3910, "num_data_blocks": 283, "num_entries": 567, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:10:17.546559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 907] Level-0 flush table #10488: 2249544 bytes OK
>2018-07-11 10:10:17.580036 7f29149e4700  1 mon.magna102@0(leader).log v20612 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20612)
>2018-07-11 10:10:17.580084 7f29149e4700  1 mon.magna102@0(leader).log v20612 check_sub sending message to client.? 10.8.128.102:0/3575177126 with 1 entries (version 20612)
>2018-07-11 10:10:17.613245 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.579728) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10488 started
>2018-07-11 10:10:17.613254 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.613208) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10488: memtable #1 done
>2018-07-11 10:10:17.613256 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.613223) EVENT_LOG_v1 {"time_micros": 1531303817613217, "job": 907, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:10:17.613258 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.613235) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:10:17.613265 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 907] Try to delete WAL files size 3839847, prev total WAL file size 3859165, number of live WAL files 2.
>
>2018-07-11 10:10:17.614160 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:10:17.614370 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 10:10:17.614377 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 65844.5 total, 1000.5 interval
>Cumulative writes: 89K writes, 422K keys, 89K commit groups, 1.0 writes per commit group, ingest: 0.70 GB, 0.01 MB/s
>Cumulative WAL: 89K writes, 89K syncs, 1.00 writes per sync, written: 0.70 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1082 writes, 4818 keys, 1082 commit groups, 1.0 writes per commit group, ingest: 7.86 MB, 0.01 MB/s
>Interval WAL: 1082 writes, 1082 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        37       454    0.081       0      0
>  L1      1/1   13.64 MB   0.0      7.1     0.6      6.5       6.5     -0.0       0.0  10.8     72.6     66.4       101       452    0.223   1614K   193K
> Sum      2/2   15.78 MB   0.0      7.1     0.6      6.5       7.1      0.6       0.0  11.7     53.1     53.1       138       906    0.152   1614K   193K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  12.9     52.0     52.2         1         8    0.149     14K   1635
>Uptime(secs): 65844.5 total, 65844.5 interval
>Flush(GB): cumulative 0.609, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.14 GB write, 0.11 MB/s write, 7.14 GB read, 0.11 MB/s read, 137.5 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     16.9        37       454    0.081       0      0
>  L1      1/1   13.64 MB   0.0      7.1     0.6      6.5       6.5     -0.0       0.0  10.8     72.6     66.4       101       452    0.223   1614K   193K
> Sum      2/2   15.78 MB   0.0      7.1     0.6      6.5       7.1      0.6       0.0  11.7     53.1     53.1       138       906    0.152   1614K   193K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 65844.5 total, 65844.5 interval
>Flush(GB): cumulative 0.609, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.14 GB write, 0.11 MB/s write, 7.14 GB read, 0.11 MB/s read, 137.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 10:10:17.614397 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.614391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:10:17.614402 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 908] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:10:17.614407 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 907 Base level 0, inputs: [10488(2196KB)], [10486(13MB)]
>
>2018-07-11 10:10:17.614420 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817614411, "job": 908, "event": "compaction_started", "files_L0": [10488], "files_L1": [10486], "score": -1, "input_data_size": 16548736}
>2018-07-11 10:10:17.805611 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 908] Generated table #10489: 3376 keys, 16284315 bytes
>2018-07-11 10:10:17.805660 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817805632, "cf_name": "default", "job": 908, "event": "table_file_creation", "file_number": 10489, "file_size": 16284315, "table_properties": {"data_size": 16159958, "index_size": 70948, "filter_size": 52482, "raw_key_size": 67882, "raw_average_key_size": 20, "raw_value_size": 16062285, "raw_average_value_size": 4757, "num_data_blocks": 2082, "num_entries": 3376, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:10:17.838804 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 908] Compacted 1@0 + 1@1 files to L1 => 16284315 bytes
>2018-07-11 10:10:17.872328 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.872260) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.8 rd, 72.6 wr, level 1, files in(1, 1) out(1) MB in(2.1, 13.6) out(15.5), read-write-amplify(14.6) write-amplify(7.2) OK, records in: 3672, records dropped: 296
>
>2018-07-11 10:10:17.872337 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.872282) EVENT_LOG_v1 {"time_micros": 1531303817872272, "job": 908, "event": "compaction_finished", "compaction_time_micros": 224366, "output_level": 1, "num_output_files": 1, "total_output_size": 16284315, "num_input_records": 3672, "num_output_records": 3376, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:10:17.872806 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817872801, "job": 908, "event": "table_file_deletion", "file_number": 10488}
>2018-07-11 10:10:17.875315 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817875312, "job": 908, "event": "table_file_deletion", "file_number": 10486}
>2018-07-11 10:10:17.875565 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10490. Immutable memtables: 0.
>
>2018-07-11 10:10:17.875604 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 909] Syncing log #10487
>2018-07-11 10:10:17.889297 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.875596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:10:17.889308 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 909] Flushing memtable with next log file: 10490
>
>2018-07-11 10:10:17.889322 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817889315, "job": 909, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21656}
>2018-07-11 10:10:17.889325 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 909] Level-0 flush table #10491: started
>2018-07-11 10:10:17.931441 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817931428, "cf_name": "default", "job": 909, "event": "table_file_creation", "file_number": 10491, "file_size": 20437, "table_properties": {"data_size": 19367, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 19144, "raw_average_value_size": 2393, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:10:17.931451 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 909] Level-0 flush table #10491: 20437 bytes OK
>2018-07-11 10:10:17.981526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.948005) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10491 started
>2018-07-11 10:10:17.981534 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.981493) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10491: memtable #1 done
>2018-07-11 10:10:17.981537 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.981507) EVENT_LOG_v1 {"time_micros": 1531303817981501, "job": 909, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:10:17.981539 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.981518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:10:17.981544 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 909] Try to delete WAL files size 19318, prev total WAL file size 19318, number of live WAL files 2.
>
>2018-07-11 10:10:17.981638 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:10:17.981685 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:17.981679) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:10:17.981694 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 910] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:10:17.981700 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 909 Base level 0, inputs: [10491(19KB)], [10489(15MB)]
>
>2018-07-11 10:10:17.981713 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303817981703, "job": 910, "event": "compaction_started", "files_L0": [10491], "files_L1": [10489], "score": -1, "input_data_size": 16304752}
>2018-07-11 10:10:18.173937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 910] Generated table #10492: 3379 keys, 16303635 bytes
>2018-07-11 10:10:18.173985 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303818173957, "cf_name": "default", "job": 910, "event": "table_file_creation", "file_number": 10492, "file_size": 16303635, "table_properties": {"data_size": 16179152, "index_size": 71017, "filter_size": 52539, "raw_key_size": 67943, "raw_average_key_size": 20, "raw_value_size": 16081389, "raw_average_value_size": 4759, "num_data_blocks": 2084, "num_entries": 3379, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:10:18.215461 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 910] Compacted 1@0 + 1@1 files to L1 => 16303635 bytes
>2018-07-11 10:10:18.248998 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:18.248929) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.8 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.5) out(15.5), read-write-amplify(1595.6) write-amplify(797.8) OK, records in: 3384, records dropped: 5
>
>2018-07-11 10:10:18.249007 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:18.248951) EVENT_LOG_v1 {"time_micros": 1531303818248941, "job": 910, "event": "compaction_finished", "compaction_time_micros": 233731, "output_level": 1, "num_output_files": 1, "total_output_size": 16303635, "num_input_records": 3384, "num_output_records": 3379, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:10:18.249084 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303818249080, "job": 910, "event": "table_file_deletion", "file_number": 10491}
>2018-07-11 10:10:18.251958 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303818251956, "job": 910, "event": "table_file_deletion", "file_number": 10489}
>2018-07-11 10:10:20.476320 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10493. Immutable memtables: 0.
>
>2018-07-11 10:10:20.476366 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 911] Syncing log #10490
>2018-07-11 10:10:20.492911 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.476356) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:10:20.492922 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 911] Flushing memtable with next log file: 10493
>
>2018-07-11 10:10:20.492949 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820492928, "job": 911, "event": "flush_started", "num_memtables": 1, "num_entries": 270, "num_deletes": 251, "memory_usage": 50664}
>2018-07-11 10:10:20.492952 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 911] Level-0 flush table #10494: started
>2018-07-11 10:10:20.535079 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820535066, "cf_name": "default", "job": 911, "event": "table_file_creation", "file_number": 10494, "file_size": 43736, "table_properties": {"data_size": 41867, "index_size": 180, "filter_size": 770, "raw_key_size": 5280, "raw_average_key_size": 20, "raw_value_size": 38252, "raw_average_value_size": 146, "num_data_blocks": 6, "num_entries": 262, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:10:20.535089 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 911] Level-0 flush table #10494: 43736 bytes OK
>2018-07-11 10:10:20.593365 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.551615) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10494 started
>2018-07-11 10:10:20.593375 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.593327) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10494: memtable #1 done
>2018-07-11 10:10:20.593378 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.593343) EVENT_LOG_v1 {"time_micros": 1531303820593336, "job": 911, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:10:20.593380 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.593355) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:10:20.593385 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 911] Try to delete WAL files size 43011, prev total WAL file size 43011, number of live WAL files 2.
>
>2018-07-11 10:10:20.593497 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:10:20.593562 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.593555) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:10:20.593574 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 912] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:10:20.593581 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 911 Base level 0, inputs: [10494(42KB)], [10492(15MB)]
>
>2018-07-11 10:10:20.593594 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820593584, "job": 912, "event": "compaction_started", "files_L0": [10494], "files_L1": [10492], "score": -1, "input_data_size": 16347371}
>2018-07-11 10:10:20.777474 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 912] Generated table #10495: 3133 keys, 14433243 bytes
>2018-07-11 10:10:20.777505 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820777491, "cf_name": "default", "job": 912, "event": "table_file_creation", "file_number": 10495, "file_size": 14433243, "table_properties": {"data_size": 14322684, "index_size": 62931, "filter_size": 46701, "raw_key_size": 63027, "raw_average_key_size": 20, "raw_value_size": 14234019, "raw_average_value_size": 4543, "num_data_blocks": 1838, "num_entries": 3133, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:10:20.819011 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 912] Compacted 1@0 + 1@1 files to L1 => 14433243 bytes
>2018-07-11 10:10:20.860884 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.860816) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 72.5 rd, 64.0 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.5) out(13.8), read-write-amplify(703.8) write-amplify(330.0) OK, records in: 3641, records dropped: 508
>
>2018-07-11 10:10:20.860893 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:10:20.860838) EVENT_LOG_v1 {"time_micros": 1531303820860829, "job": 912, "event": "compaction_finished", "compaction_time_micros": 225402, "output_level": 1, "num_output_files": 1, "total_output_size": 14433243, "num_input_records": 3641, "num_output_records": 3133, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:10:20.860989 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820860985, "job": 912, "event": "table_file_deletion", "file_number": 10494}
>2018-07-11 10:10:20.863903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531303820863900, "job": 912, "event": "table_file_deletion", "file_number": 10492}
>2018-07-11 10:10:21.188995 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:10:21.189018 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/910603743' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:10:21.544732 7f29149e4700  1 mon.magna102@0(leader).log v20613 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20613)
>2018-07-11 10:10:22.544699 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:10:22.695208 7f29149e4700  1 mon.magna102@0(leader).log v20614 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20614)
>2018-07-11 10:10:25.024347 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:10:25.024371 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1461891831' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:10:25.478848 7f29149e4700  1 mon.magna102@0(leader).log v20615 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20615)
>2018-07-11 10:11:02.360213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 5,10 (REQUEST_SLOW)
>2018-07-11 10:11:02.515339 7f29149e4700  1 mon.magna102@0(leader).log v20616 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20616)
>2018-07-11 10:11:02.838483 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:11:14.365516 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:11:14.517643 7f29149e4700  1 mon.magna102@0(leader).log v20617 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20617)
>2018-07-11 10:11:22.369064 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:11:22.526684 7f29149e4700  1 mon.magna102@0(leader).log v20618 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20618)
>2018-07-11 10:11:32.373654 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:11:32.544234 7f29149e4700  1 mon.magna102@0(leader).log v20619 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20619)
>2018-07-11 10:11:34.094347 7f29149e4700  1 mon.magna102@0(leader).log v20620 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20620)
>2018-07-11 10:12:02.838620 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:12:06.388781 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:06.563690 7f29149e4700  1 mon.magna102@0(leader).log v20621 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20621)
>2018-07-11 10:12:12.391372 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:12.540486 7f29149e4700  1 mon.magna102@0(leader).log v20622 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20622)
>2018-07-11 10:12:17.430142 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 44 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:17.540949 7f29149e4700  1 mon.magna102@0(leader).log v20623 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20623)
>2018-07-11 10:12:26.397628 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:26.550203 7f29149e4700  1 mon.magna102@0(leader).log v20624 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20624)
>2018-07-11 10:12:32.400280 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 56 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:32.550692 7f29149e4700  1 mon.magna102@0(leader).log v20625 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20625)
>2018-07-11 10:12:37.433965 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:37.534552 7f29149e4700  1 mon.magna102@0(leader).log v20626 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20626)
>2018-07-11 10:12:46.406510 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 62 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:46.560463 7f29149e4700  1 mon.magna102@0(leader).log v20627 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20627)
>2018-07-11 10:12:52.409230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 68 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:12:52.577828 7f29149e4700  1 mon.magna102@0(leader).log v20628 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20628)
>2018-07-11 10:13:02.413615 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 68 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:13:02.570335 7f29149e4700  1 mon.magna102@0(leader).log v20629 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20629)
>2018-07-11 10:13:02.838795 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:13:18.420501 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 10:13:18.589615 7f29149e4700  1 mon.magna102@0(leader).log v20630 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20630)
>2018-07-11 10:13:26.424130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 12 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:13:26.606992 7f29149e4700  1 mon.magna102@0(leader).log v20631 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20631)
>2018-07-11 10:13:32.426700 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:13:32.599267 7f29149e4700  1 mon.magna102@0(leader).log v20632 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20632)
>2018-07-11 10:13:42.431206 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:13:42.583463 7f29149e4700  1 mon.magna102@0(leader).log v20633 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20633)
>2018-07-11 10:13:47.466686 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:13:47.567244 7f29149e4700  1 mon.magna102@0(leader).log v20634 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20634)
>2018-07-11 10:13:57.467098 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:13:57.568121 7f29149e4700  1 mon.magna102@0(leader).log v20635 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20635)
>2018-07-11 10:14:02.838967 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:14:32.453278 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 5,10 (REQUEST_SLOW)
>2018-07-11 10:14:32.597007 7f29149e4700  1 mon.magna102@0(leader).log v20636 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20636)
>2018-07-11 10:14:43.940888 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:14:43.940912 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3177022294' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:14:44.064508 7f29149e4700  1 mon.magna102@0(leader).log v20637 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20637)
>2018-07-11 10:14:45.064497 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:14:45.214867 7f29149e4700  1 mon.magna102@0(leader).log v20638 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20638)
>2018-07-11 10:14:46.146773 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:14:46.146798 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1769590255' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:14:46.148152 7f29191ed700  1 mon.magna102@0(leader).log v20638 check_sub sending message to client.? 10.8.128.102:0/1769590255 with 1 entries (version 20638)
>2018-07-11 10:14:46.281685 7f29149e4700  1 mon.magna102@0(leader).log v20639 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20639)
>2018-07-11 10:14:46.281719 7f29149e4700  1 mon.magna102@0(leader).log v20639 check_sub sending message to client.? 10.8.128.102:0/1769590255 with 0 entries (version 20639)
>2018-07-11 10:14:52.462638 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:14:52.615863 7f29149e4700  1 mon.magna102@0(leader).log v20640 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20640)
>2018-07-11 10:15:02.467135 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:15:02.625046 7f29149e4700  1 mon.magna102@0(leader).log v20641 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20641)
>2018-07-11 10:15:02.839107 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:15:42.484920 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:15:42.637673 7f29149e4700  1 mon.magna102@0(leader).log v20642 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20642)
>2018-07-11 10:15:47.526951 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:15:47.621432 7f29149e4700  1 mon.magna102@0(leader).log v20643 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20643)
>2018-07-11 10:15:56.491173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:15:56.639003 7f29149e4700  1 mon.magna102@0(leader).log v20644 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20644)
>2018-07-11 10:16:02.493869 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 46 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:16:02.647917 7f29149e4700  1 mon.magna102@0(leader).log v20645 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20645)
>2018-07-11 10:16:02.839235 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:16:07.531148 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 52 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:16:07.631702 7f29149e4700  1 mon.magna102@0(leader).log v20646 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20646)
>2018-07-11 10:16:16.500102 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 64 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:16:16.659050 7f29149e4700  1 mon.magna102@0(leader).log v20647 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20647)
>2018-07-11 10:16:22.502777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 68 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:16:22.659642 7f29149e4700  1 mon.magna102@0(leader).log v20648 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20648)
>2018-07-11 10:16:27.542894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 72 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:16:27.643420 7f29149e4700  1 mon.magna102@0(leader).log v20649 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20649)
>2018-07-11 10:16:36.509057 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 74 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:16:36.660987 7f29149e4700  1 mon.magna102@0(leader).log v20650 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20650)
>2018-07-11 10:16:50.515243 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 10:16:50.670541 7f29149e4700  1 mon.magna102@0(leader).log v20651 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20651)
>2018-07-11 10:16:56.517805 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:16:56.729470 7f29149e4700  1 mon.magna102@0(leader).log v20652 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20652)
>2018-07-11 10:17:02.520423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:17:02.688385 7f29149e4700  1 mon.magna102@0(leader).log v20653 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20653)
>2018-07-11 10:17:02.839409 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8791 MB, avail 861 GB
>2018-07-11 10:17:07.562757 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:17:07.680392 7f29149e4700  1 mon.magna102@0(leader).log v20654 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20654)
>2018-07-11 10:17:16.526631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:17:16.666172 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10496. Immutable memtables: 0.
>
>2018-07-11 10:17:16.699513 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.666257) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:17:16.699526 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 913] Flushing memtable with next log file: 10496
>
>2018-07-11 10:17:16.699541 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304236699532, "job": 913, "event": "flush_started", "num_memtables": 1, "num_entries": 2048, "num_deletes": 251, "memory_usage": 4061824}
>2018-07-11 10:17:16.699546 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 913] Level-0 flush table #10497: started
>2018-07-11 10:17:16.778587 7f29149e4700  1 mon.magna102@0(leader).log v20655 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20655)
>2018-07-11 10:17:16.800097 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304236800069, "cf_name": "default", "job": 913, "event": "table_file_creation", "file_number": 10497, "file_size": 3869605, "table_properties": {"data_size": 3839115, "index_size": 16809, "filter_size": 12755, "raw_key_size": 16545, "raw_average_key_size": 20, "raw_value_size": 3815944, "raw_average_value_size": 4758, "num_data_blocks": 504, "num_entries": 802, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:17:16.800108 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 913] Level-0 flush table #10497: 3869605 bytes OK
>2018-07-11 10:17:16.850156 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.816694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10497 started
>2018-07-11 10:17:16.850164 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.850117) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10497: memtable #1 done
>2018-07-11 10:17:16.850167 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.850131) EVENT_LOG_v1 {"time_micros": 1531304236850125, "job": 913, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:17:16.850189 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.850145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:17:16.850196 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 913] Try to delete WAL files size 4021993, prev total WAL file size 4041877, number of live WAL files 2.
>
>2018-07-11 10:17:16.851124 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:17:16.851200 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:17:16.851192) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:17:16.851207 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 914] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:17:16.851213 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 913 Base level 0, inputs: [10497(3778KB)], [10495(13MB)]
>
>2018-07-11 10:17:16.851226 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304236851216, "job": 914, "event": "compaction_started", "files_L0": [10497], "files_L1": [10495], "score": -1, "input_data_size": 18302848}
>2018-07-11 10:17:17.042573 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 914] Generated table #10498: 3425 keys, 16400251 bytes
>2018-07-11 10:17:17.042618 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304237042590, "cf_name": "default", "job": 914, "event": "table_file_creation", "file_number": 10498, "file_size": 16400251, "table_properties": {"data_size": 16274697, "index_size": 71756, "filter_size": 52871, "raw_key_size": 69325, "raw_average_key_size": 20, "raw_value_size": 16175620, "raw_average_value_size": 4722, "num_data_blocks": 2090, "num_entries": 3425, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:17:17.075757 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 914] Compacted 1@0 + 1@1 files to L1 => 16400251 bytes
>2018-07-11 10:17:17.117601 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:17:17.117535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.5 rd, 73.0 wr, level 1, files in(1, 1) out(1) MB in(3.7, 13.8) out(15.6), read-write-amplify(9.0) write-amplify(4.2) OK, records in: 3935, records dropped: 510
>
>2018-07-11 10:17:17.117610 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:17:17.117556) EVENT_LOG_v1 {"time_micros": 1531304237117547, "job": 914, "event": "compaction_finished", "compaction_time_micros": 224514, "output_level": 1, "num_output_files": 1, "total_output_size": 16400251, "num_input_records": 3935, "num_output_records": 3425, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:17:17.118471 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304237118466, "job": 914, "event": "table_file_deletion", "file_number": 10497}
>2018-07-11 10:17:17.121317 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304237121314, "job": 914, "event": "table_file_deletion", "file_number": 10495}
>2018-07-11 10:17:58.615373 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:17:58.615396 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/960871783' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:17:59.671205 7f29149e4700  1 mon.magna102@0(leader).log v20656 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20656)
>2018-07-11 10:18:00.067641 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:18:00.067663 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1232992642' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:18:00.804880 7f29149e4700  1 mon.magna102@0(leader).log v20657 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20657)
>2018-07-11 10:18:02.111235 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:18:02.111258 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2376860917' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:18:02.230078 7f29149e4700  1 mon.magna102@0(leader).log v20658 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20658)
>2018-07-11 10:18:02.839600 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:18:03.807890 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:18:03.807915 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2237294079' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:18:03.809215 7f29191ed700  1 mon.magna102@0(leader).log v20658 check_sub sending message to client.? 10.8.128.102:0/2237294079 with 1 entries (version 20658)
>2018-07-11 10:18:04.363737 7f29149e4700  1 mon.magna102@0(leader).log v20659 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20659)
>2018-07-11 10:18:04.363770 7f29149e4700  1 mon.magna102@0(leader).log v20659 check_sub sending message to client.? 10.8.128.102:0/2237294079 with 0 entries (version 20659)
>2018-07-11 10:18:08.550934 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 21 slow requests are blocked > 32 sec. Implicated osds 5,10 (REQUEST_SLOW)
>2018-07-11 10:18:08.707357 7f29149e4700  1 mon.magna102@0(leader).log v20660 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20660)
>2018-07-11 10:18:14.553620 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:18:14.716243 7f29149e4700  1 mon.magna102@0(leader).log v20661 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20661)
>2018-07-11 10:18:22.557066 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 21 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:18:22.725385 7f29149e4700  1 mon.magna102@0(leader).log v20662 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20662)
>2018-07-11 10:18:32.561404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:18:32.734736 7f29149e4700  1 mon.magna102@0(leader).log v20663 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20663)
>2018-07-11 10:18:37.693698 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10499. Immutable memtables: 0.
>
>2018-07-11 10:18:37.693742 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 915] Syncing log #10496
>2018-07-11 10:18:37.710244 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.693732) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:18:37.710256 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 915] Flushing memtable with next log file: 10499
>
>2018-07-11 10:18:37.710282 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304317710274, "job": 915, "event": "flush_started", "num_memtables": 1, "num_entries": 607, "num_deletes": 250, "memory_usage": 819016}
>2018-07-11 10:18:37.710285 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 915] Level-0 flush table #10500: started
>2018-07-11 10:18:37.752405 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304317752391, "cf_name": "default", "job": 915, "event": "table_file_creation", "file_number": 10500, "file_size": 544828, "table_properties": {"data_size": 539486, "index_size": 2123, "filter_size": 2300, "raw_key_size": 7210, "raw_average_key_size": 21, "raw_value_size": 533425, "raw_average_value_size": 1606, "num_data_blocks": 66, "num_entries": 332, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:18:37.752415 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 915] Level-0 flush table #10500: 544828 bytes OK
>2018-07-11 10:18:37.802462 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.768965) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10500 started
>2018-07-11 10:18:37.802470 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.802427) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10500: memtable #1 done
>2018-07-11 10:18:37.802473 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.802441) EVENT_LOG_v1 {"time_micros": 1531304317802435, "job": 915, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:18:37.802488 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.802453) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:18:37.802495 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 915] Try to delete WAL files size 805637, prev total WAL file size 805637, number of live WAL files 2.
>
>2018-07-11 10:18:37.802786 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:18:37.802838 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:18:37.802830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:18:37.802847 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 916] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:18:37.802853 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 915 Base level 0, inputs: [10500(532KB)], [10498(15MB)]
>
>2018-07-11 10:18:37.802866 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304317802857, "job": 916, "event": "compaction_started", "files_L0": [10500], "files_L1": [10498], "score": -1, "input_data_size": 16945079}
>2018-07-11 10:18:37.986507 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 916] Generated table #10501: 3283 keys, 15458190 bytes
>2018-07-11 10:18:37.986553 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304317986527, "cf_name": "default", "job": 916, "event": "table_file_creation", "file_number": 10501, "file_size": 15458190, "table_properties": {"data_size": 15341118, "index_size": 66428, "filter_size": 49717, "raw_key_size": 66074, "raw_average_key_size": 20, "raw_value_size": 15247867, "raw_average_value_size": 4644, "num_data_blocks": 1940, "num_entries": 3283, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:18:38.019693 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 916] Compacted 1@0 + 1@1 files to L1 => 15458190 bytes
>2018-07-11 10:18:38.053255 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:18:38.053185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.2 rd, 71.3 wr, level 1, files in(1, 1) out(1) MB in(0.5, 15.6) out(14.7), read-write-amplify(59.5) write-amplify(28.4) OK, records in: 3757, records dropped: 474
>
>2018-07-11 10:18:38.053264 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:18:38.053207) EVENT_LOG_v1 {"time_micros": 1531304318053198, "job": 916, "event": "compaction_finished", "compaction_time_micros": 216811, "output_level": 1, "num_output_files": 1, "total_output_size": 15458190, "num_input_records": 3757, "num_output_records": 3283, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:18:38.053443 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304318053438, "job": 916, "event": "table_file_deletion", "file_number": 10500}
>2018-07-11 10:18:38.056420 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304318056417, "job": 916, "event": "table_file_deletion", "file_number": 10498}
>2018-07-11 10:18:38.056537 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:19:02.839779 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:19:06.576502 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:06.762019 7f29149e4700  1 mon.magna102@0(leader).log v20664 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20664)
>2018-07-11 10:19:16.580957 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 26 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:16.749917 7f29149e4700  1 mon.magna102@0(leader).log v20665 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20665)
>2018-07-11 10:19:22.583509 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 28 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:22.758836 7f29149e4700  1 mon.magna102@0(leader).log v20666 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20666)
>2018-07-11 10:19:27.657618 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 32 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:27.775943 7f29149e4700  1 mon.magna102@0(leader).log v20667 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20667)
>2018-07-11 10:19:32.657849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:33.718249 7f29149e4700  1 mon.magna102@0(leader).log v20668 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20668)
>2018-07-11 10:19:37.658061 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:37.768813 7f29149e4700  1 mon.magna102@0(leader).log v20669 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20669)
>2018-07-11 10:19:46.594131 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:46.744652 7f29149e4700  1 mon.magna102@0(leader).log v20670 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20670)
>2018-07-11 10:19:52.596681 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 48 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:52.746540 7f29149e4700  1 mon.magna102@0(leader).log v20671 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20671)
>2018-07-11 10:19:57.658805 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 52 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:19:57.805516 7f29149e4700  1 mon.magna102@0(leader).log v20672 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20672)
>2018-07-11 10:20:02.688682 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:20:02.839957 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:20:03.739777 7f29149e4700  1 mon.magna102@0(leader).log v20673 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20673)
>2018-07-11 10:20:06.603153 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 54 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:20:06.756935 7f29149e4700  1 mon.magna102@0(leader).log v20674 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20674)
>2018-07-11 10:20:22.610023 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 10:20:22.766637 7f29149e4700  1 mon.magna102@0(leader).log v20675 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20675)
>2018-07-11 10:20:27.689649 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 10:20:27.808724 7f29149e4700  1 mon.magna102@0(leader).log v20676 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20676)
>2018-07-11 10:20:36.616090 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:20:36.768243 7f29149e4700  1 mon.magna102@0(leader).log v20677 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20677)
>2018-07-11 10:20:42.618749 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:20:42.785485 7f29149e4700  1 mon.magna102@0(leader).log v20678 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20678)
>2018-07-11 10:20:47.690417 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 12 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:20:47.787159 7f29149e4700  1 mon.magna102@0(leader).log v20679 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20679)
>2018-07-11 10:20:49.998441 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:20:49.998464 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/265872117' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:20:50.129200 7f29149e4700  1 mon.magna102@0(leader).log v20680 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20680)
>2018-07-11 10:20:52.690649 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:20:53.754687 7f29149e4700  1 mon.magna102@0(leader).log v20681 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20681)
>2018-07-11 10:20:57.690912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:20:57.821932 7f29149e4700  1 mon.magna102@0(leader).log v20682 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20682)
>2018-07-11 10:21:02.840138 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:21:42.645227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 17 slow requests are blocked > 32 sec. Implicated osds 5,10 (REQUEST_SLOW)
>2018-07-11 10:21:42.800267 7f29149e4700  1 mon.magna102@0(leader).log v20683 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20683)
>2018-07-11 10:21:48.647914 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:21:48.802074 7f29149e4700  1 mon.magna102@0(leader).log v20684 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20684)
>2018-07-11 10:22:02.654057 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 17 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:02.803221 7f29149e4700  1 mon.magna102@0(leader).log v20685 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20685)
>2018-07-11 10:22:02.840337 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:22:07.693400 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:07.803664 7f29149e4700  1 mon.magna102@0(leader).log v20686 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20686)
>2018-07-11 10:22:36.669070 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:36.839340 7f29149e4700  1 mon.magna102@0(leader).log v20687 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20687)
>2018-07-11 10:22:42.671619 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:42.848349 7f29149e4700  1 mon.magna102@0(leader).log v20688 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20688)
>2018-07-11 10:22:47.714810 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 30 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:47.840457 7f29149e4700  1 mon.magna102@0(leader).log v20689 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20689)
>2018-07-11 10:22:57.717223 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:22:57.809267 7f29149e4700  1 mon.magna102@0(leader).log v20690 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20690)
>2018-07-11 10:23:02.840519 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:23:06.682269 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:06.835162 7f29149e4700  1 mon.magna102@0(leader).log v20691 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20691)
>2018-07-11 10:23:12.684735 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:12.827507 7f29149e4700  1 mon.magna102@0(leader).log v20692 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20692)
>2018-07-11 10:23:17.717971 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 46 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:17.827895 7f29149e4700  1 mon.magna102@0(leader).log v20693 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20693)
>2018-07-11 10:23:26.691019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 52 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:26.845539 7f29149e4700  1 mon.magna102@0(leader).log v20694 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20694)
>2018-07-11 10:23:31.699609 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:23:31.699635 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3664440844' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:23:31.700983 7f29191ed700  1 mon.magna102@0(leader).log v20694 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20694)
>2018-07-11 10:23:31.829317 7f29149e4700  1 mon.magna102@0(leader).log v20695 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20695)
>2018-07-11 10:23:31.829350 7f29149e4700  1 mon.magna102@0(leader).log v20695 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 0 entries (version 20695)
>2018-07-11 10:23:36.695483 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 60 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:36.854956 7f29149e4700  1 mon.magna102@0(leader).log v20696 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20696)
>2018-07-11 10:23:36.855001 7f29149e4700  1 mon.magna102@0(leader).log v20696 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20696)
>2018-07-11 10:23:46.699917 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 68 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:23:46.855813 7f29149e4700  1 mon.magna102@0(leader).log v20697 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20697)
>2018-07-11 10:23:46.855859 7f29149e4700  1 mon.magna102@0(leader).log v20697 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20697)
>2018-07-11 10:23:52.702500 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 68 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:23:52.874400 7f29149e4700  1 mon.magna102@0(leader).log v20698 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20698)
>2018-07-11 10:23:52.874445 7f29149e4700  1 mon.magna102@0(leader).log v20698 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20698)
>2018-07-11 10:24:02.706906 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 10:24:02.840695 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:24:02.858594 7f29149e4700  1 mon.magna102@0(leader).log v20699 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20699)
>2018-07-11 10:24:02.858639 7f29149e4700  1 mon.magna102@0(leader).log v20699 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20699)
>2018-07-11 10:24:02.933981 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10502. Immutable memtables: 0.
>
>2018-07-11 10:24:02.934026 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 917] Syncing log #10499
>2018-07-11 10:24:02.950562 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:24:02.934017) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:24:02.950573 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 917] Flushing memtable with next log file: 10502
>
>2018-07-11 10:24:02.950587 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304642950580, "job": 917, "event": "flush_started", "num_memtables": 1, "num_entries": 1692, "num_deletes": 251, "memory_usage": 3277016}
>2018-07-11 10:24:02.950590 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 917] Level-0 flush table #10503: started
>2018-07-11 10:24:03.017787 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304643017772, "cf_name": "default", "job": 917, "event": "table_file_creation", "file_number": 10503, "file_size": 3123095, "table_properties": {"data_size": 3098433, "index_size": 13356, "filter_size": 10382, "raw_key_size": 14379, "raw_average_key_size": 20, "raw_value_size": 3079142, "raw_average_value_size": 4411, "num_data_blocks": 402, "num_entries": 698, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:24:03.017811 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 917] Level-0 flush table #10503: 3123095 bytes OK
>2018-07-11 10:24:03.076145 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.034298) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10503 started
>2018-07-11 10:24:03.076153 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.076107) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10503: memtable #1 done
>2018-07-11 10:24:03.076156 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.076121) EVENT_LOG_v1 {"time_micros": 1531304643076115, "job": 917, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:24:03.076158 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.076135) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:24:03.076165 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 917] Try to delete WAL files size 3243872, prev total WAL file size 3243872, number of live WAL files 2.
>
>2018-07-11 10:24:03.076897 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:24:03.077073 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 10:24:03.077080 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 66669.9 total, 825.5 interval
>Cumulative writes: 90K writes, 427K keys, 90K commit groups, 1.0 writes per commit group, ingest: 0.71 GB, 0.01 MB/s
>Cumulative WAL: 90K writes, 90K syncs, 1.00 writes per sync, written: 0.71 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1010 writes, 4617 keys, 1010 commit groups, 1.0 writes per commit group, ingest: 7.74 MB, 0.01 MB/s
>Interval WAL: 1010 writes, 1010 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.98 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        37       459    0.081       0      0
>  L1      1/1   14.74 MB   0.0      7.2     0.6      6.6       6.6     -0.0       0.0  10.8     72.5     66.4       102       457    0.223   1632K   194K
> Sum      2/2   17.72 MB   0.0      7.2     0.6      6.6       7.2      0.6       0.0  11.7     53.1     53.2       139       916    0.152   1632K   194K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.4     53.6     54.9         2        10    0.150     18K   1793
>Uptime(secs): 66669.9 total, 66669.9 interval
>Flush(GB): cumulative 0.616, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.22 GB write, 0.11 MB/s write, 7.21 GB read, 0.11 MB/s read, 139.0 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.98 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        37       459    0.081       0      0
>  L1      1/1   14.74 MB   0.0      7.2     0.6      6.6       6.6     -0.0       0.0  10.8     72.5     66.4       102       457    0.223   1632K   194K
> Sum      2/2   17.72 MB   0.0      7.2     0.6      6.6       7.2      0.6       0.0  11.7     53.1     53.2       139       916    0.152   1632K   194K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 66669.9 total, 66669.9 interval
>Flush(GB): cumulative 0.616, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.22 GB write, 0.11 MB/s write, 7.21 GB read, 0.11 MB/s read, 139.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 10:24:03.077103 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.077094) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:24:03.077110 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 918] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:24:03.077116 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 917 Base level 0, inputs: [10503(3049KB)], [10501(14MB)]
>
>2018-07-11 10:24:03.077127 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304643077119, "job": 918, "event": "compaction_started", "files_L0": [10503], "files_L1": [10501], "score": -1, "input_data_size": 18581285}
>2018-07-11 10:24:03.268533 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 918] Generated table #10504: 3469 keys, 16600911 bytes
>2018-07-11 10:24:03.268581 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304643268554, "cf_name": "default", "job": 918, "event": "table_file_creation", "file_number": 10504, "file_size": 16600911, "table_properties": {"data_size": 16474867, "index_size": 71763, "filter_size": 53354, "raw_key_size": 70152, "raw_average_key_size": 20, "raw_value_size": 16375161, "raw_average_value_size": 4720, "num_data_blocks": 2089, "num_entries": 3469, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:24:03.301694 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 918] Compacted 1@0 + 1@1 files to L1 => 16600911 bytes
>2018-07-11 10:24:03.335227 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.335152) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.7 rd, 73.9 wr, level 1, files in(1, 1) out(1) MB in(3.0, 14.7) out(15.8), read-write-amplify(11.3) write-amplify(5.3) OK, records in: 3981, records dropped: 512
>
>2018-07-11 10:24:03.335236 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:24:03.335173) EVENT_LOG_v1 {"time_micros": 1531304643335163, "job": 918, "event": "compaction_finished", "compaction_time_micros": 224551, "output_level": 1, "num_output_files": 1, "total_output_size": 16600911, "num_input_records": 3981, "num_output_records": 3469, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:24:03.335832 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304643335828, "job": 918, "event": "table_file_deletion", "file_number": 10503}
>2018-07-11 10:24:03.338870 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304643338867, "job": 918, "event": "table_file_deletion", "file_number": 10501}
>2018-07-11 10:24:07.749556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:24:07.852777 7f29149e4700  1 mon.magna102@0(leader).log v20700 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20700)
>2018-07-11 10:24:07.852822 7f29149e4700  1 mon.magna102@0(leader).log v20700 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20700)
>2018-07-11 10:24:16.713124 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:24:16.887019 7f29149e4700  1 mon.magna102@0(leader).log v20701 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20701)
>2018-07-11 10:24:16.887065 7f29149e4700  1 mon.magna102@0(leader).log v20701 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20701)
>2018-07-11 10:24:26.717568 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:24:26.887958 7f29149e4700  1 mon.magna102@0(leader).log v20702 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20702)
>2018-07-11 10:24:26.888003 7f29149e4700  1 mon.magna102@0(leader).log v20702 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20702)
>2018-07-11 10:24:28.788173 7f29149e4700  1 mon.magna102@0(leader).log v20703 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20703)
>2018-07-11 10:24:28.788214 7f29149e4700  1 mon.magna102@0(leader).log v20703 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 0 entries (version 20703)
>2018-07-11 10:24:32.720269 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 10:24:32.873457 7f29149e4700  1 mon.magna102@0(leader).log v20704 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20704)
>2018-07-11 10:24:32.873503 7f29149e4700  1 mon.magna102@0(leader).log v20704 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20704)
>2018-07-11 10:24:50.728382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 0,2,5,8,10 (REQUEST_SLOW)
>2018-07-11 10:24:50.891504 7f29149e4700  1 mon.magna102@0(leader).log v20705 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20705)
>2018-07-11 10:24:50.891550 7f29149e4700  1 mon.magna102@0(leader).log v20705 check_sub sending message to client.? 10.8.128.102:0/3664440844 with 1 entries (version 20705)
>2018-07-11 10:25:02.840888 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:25:18.740724 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,5,8,10 (REQUEST_SLOW)
>2018-07-11 10:25:18.893604 7f29149e4700  1 mon.magna102@0(leader).log v20706 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20706)
>2018-07-11 10:25:20.642570 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:25:20.642593 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3486259720' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:25:20.760382 7f29149e4700  1 mon.magna102@0(leader).log v20707 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20707)
>2018-07-11 10:25:28.745424 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:25:28.894671 7f29149e4700  1 mon.magna102@0(leader).log v20708 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20708)
>2018-07-11 10:25:36.748978 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:25:36.903738 7f29149e4700  1 mon.magna102@0(leader).log v20709 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20709)
>2018-07-11 10:25:46.753360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:25:46.922592 7f29149e4700  1 mon.magna102@0(leader).log v20710 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20710)
>2018-07-11 10:26:00.759630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,10 (REQUEST_SLOW)
>2018-07-11 10:26:00.907045 7f29149e4700  1 mon.magna102@0(leader).log v20711 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20711)
>2018-07-11 10:26:02.841064 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:26:07.798314 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:26:07.882722 7f29149e4700  1 mon.magna102@0(leader).log v20712 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20712)
>2018-07-11 10:26:20.768487 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 8,10 (REQUEST_SLOW)
>2018-07-11 10:26:20.917247 7f29149e4700  1 mon.magna102@0(leader).log v20713 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20713)
>2018-07-11 10:26:27.809067 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:26:27.901145 7f29149e4700  1 mon.magna102@0(leader).log v20714 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20714)
>2018-07-11 10:26:57.879691 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10505. Immutable memtables: 0.
>
>2018-07-11 10:26:57.879735 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 919] Syncing log #10502
>2018-07-11 10:26:57.896336 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.879725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:26:57.896348 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 919] Flushing memtable with next log file: 10505
>
>2018-07-11 10:26:57.896374 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304817896366, "job": 919, "event": "flush_started", "num_memtables": 1, "num_entries": 984, "num_deletes": 250, "memory_usage": 1678232}
>2018-07-11 10:26:57.896377 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 919] Level-0 flush table #10506: started
>2018-07-11 10:26:57.946872 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304817946858, "cf_name": "default", "job": 919, "event": "table_file_creation", "file_number": 10506, "file_size": 1035581, "table_properties": {"data_size": 1026812, "index_size": 4046, "filter_size": 3804, "raw_key_size": 8516, "raw_average_key_size": 21, "raw_value_size": 1018458, "raw_average_value_size": 2565, "num_data_blocks": 126, "num_entries": 397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:26:57.946882 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 919] Level-0 flush table #10506: 1035581 bytes OK
>2018-07-11 10:26:57.996905 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.963409) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10506 started
>2018-07-11 10:26:57.996913 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.996869) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10506: memtable #1 done
>2018-07-11 10:26:57.996915 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.996884) EVENT_LOG_v1 {"time_micros": 1531304817996878, "job": 919, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:26:57.996930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.996895) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:26:57.996936 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 919] Try to delete WAL files size 1658053, prev total WAL file size 1658053, number of live WAL files 2.
>
>2018-07-11 10:26:57.997396 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:26:57.997446 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:26:57.997439) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:26:57.997455 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 920] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:26:57.997462 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 919 Base level 0, inputs: [10506(1011KB)], [10504(15MB)]
>
>2018-07-11 10:26:57.997476 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304817997467, "job": 920, "event": "compaction_started", "files_L0": [10506], "files_L1": [10504], "score": -1, "input_data_size": 17636492}
>2018-07-11 10:26:58.189293 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 920] Generated table #10507: 3439 keys, 16471847 bytes
>2018-07-11 10:26:58.189342 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304818189326, "cf_name": "default", "job": 920, "event": "table_file_creation", "file_number": 10507, "file_size": 16471847, "table_properties": {"data_size": 16348150, "index_size": 70030, "filter_size": 52740, "raw_key_size": 69241, "raw_average_key_size": 20, "raw_value_size": 16250171, "raw_average_value_size": 4725, "num_data_blocks": 2045, "num_entries": 3439, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:26:58.222479 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 920] Compacted 1@0 + 1@1 files to L1 => 16471847 bytes
>2018-07-11 10:26:58.255996 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:26:58.255928) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.4 rd, 73.2 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.8) out(15.7), read-write-amplify(32.9) write-amplify(15.9) OK, records in: 3866, records dropped: 427
>
>2018-07-11 10:26:58.256005 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:26:58.255949) EVENT_LOG_v1 {"time_micros": 1531304818255940, "job": 920, "event": "compaction_finished", "compaction_time_micros": 224987, "output_level": 1, "num_output_files": 1, "total_output_size": 16471847, "num_input_records": 3866, "num_output_records": 3439, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:26:58.256292 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304818256288, "job": 920, "event": "table_file_deletion", "file_number": 10506}
>2018-07-11 10:26:58.259193 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531304818259190, "job": 920, "event": "table_file_deletion", "file_number": 10504}
>2018-07-11 10:26:58.259237 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:27:02.841248 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:27:30.799499 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,10 (REQUEST_SLOW)
>2018-07-11 10:27:30.975631 7f29149e4700  1 mon.magna102@0(leader).log v20715 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20715)
>2018-07-11 10:27:37.847613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:27:37.960864 7f29149e4700  1 mon.magna102@0(leader).log v20716 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20716)
>2018-07-11 10:27:40.803771 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:27:40.961382 7f29149e4700  1 mon.magna102@0(leader).log v20717 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20717)
>2018-07-11 10:27:42.804508 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 10:27:42.953371 7f29149e4700  1 mon.magna102@0(leader).log v20718 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20718)
>2018-07-11 10:27:47.847992 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 10:27:47.937134 7f29149e4700  1 mon.magna102@0(leader).log v20719 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20719)
>2018-07-11 10:27:52.848227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,5,8 (REQUEST_SLOW)
>2018-07-11 10:27:52.962802 7f29149e4700  1 mon.magna102@0(leader).log v20720 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20720)
>2018-07-11 10:28:00.812482 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,2,5,8 (REQUEST_SLOW)
>2018-07-11 10:28:00.963535 7f29149e4700  1 mon.magna102@0(leader).log v20721 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20721)
>2018-07-11 10:28:02.855243 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:29:02.869100 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:29:06.841295 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,5 (REQUEST_SLOW)
>2018-07-11 10:29:07.011466 7f29149e4700  1 mon.magna102@0(leader).log v20722 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20722)
>2018-07-11 10:29:12.878389 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,5 (REQUEST_SLOW)
>2018-07-11 10:29:12.995407 7f29149e4700  1 mon.magna102@0(leader).log v20723 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20723)
>2018-07-11 10:29:18.846651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:29:19.004351 7f29149e4700  1 mon.magna102@0(leader).log v20724 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20724)
>2018-07-11 10:29:27.077253 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:29:27.077274 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/584934759' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:29:27.887971 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:29:27.996736 7f29149e4700  1 mon.magna102@0(leader).log v20725 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20725)
>2018-07-11 10:29:28.627608 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:29:28.627627 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/4110857143' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:29:29.130371 7f29149e4700  1 mon.magna102@0(leader).log v20726 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20726)
>2018-07-11 10:29:32.888923 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:29:33.005913 7f29149e4700  1 mon.magna102@0(leader).log v20727 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20727)
>2018-07-11 10:29:37.889134 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:29:37.989674 7f29149e4700  1 mon.magna102@0(leader).log v20728 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20728)
>2018-07-11 10:30:02.909518 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:30:50.887027 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,10 (REQUEST_SLOW)
>2018-07-11 10:30:51.047894 7f29149e4700  1 mon.magna102@0(leader).log v20729 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20729)
>2018-07-11 10:30:57.852610 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 10:30:57.852625 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 10:30:57.939649 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:30:58.023469 7f29149e4700  1 mon.magna102@0(leader).log v20730 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20730)
>2018-07-11 10:30:58.268079 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 10:30:58.268093 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 10:30:58.465569 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 10:30:58.465583 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 10:30:58.778138 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 10:30:58.778152 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 10:30:59.090227 7f29149e4700  1 mon.magna102@0(leader).log v20731 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20731)
>2018-07-11 10:31:00.467718 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 10:31:00.467732 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 10:31:00.582329 7f29149e4700  1 mon.magna102@0(leader).log v20732 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20732)
>2018-07-11 10:31:01.615725 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 10:31:01.615742 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 10:31:01.732710 7f29149e4700  1 mon.magna102@0(leader).log v20733 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20733)
>2018-07-11 10:31:01.751552 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 10:31:01.751566 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 10:31:02.366528 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 10:31:02.366542 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 10:31:02.454448 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 10:31:02.454462 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 10:31:02.712241 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 10:31:02.712255 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 10:31:02.770157 7f29191ed700  1 mon.magna102@0(leader).osd e537 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 10:31:02.770171 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 10:31:02.799568 7f29149e4700  1 mon.magna102@0(leader).log v20734 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20734)
>2018-07-11 10:31:02.909692 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:31:02.939933 7f291b9f2700  1 mon.magna102@0(leader).osd e537  we have enough reporters to mark osd.10 down
>2018-07-11 10:31:02.939953 7f291b9f2700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (6 reporters from different host after 20.169843 >= grace 20.083915)
>2018-07-11 10:31:02.940886 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 10:31:03.016619 7f29149e4700  1 mon.magna102@0(leader).osd e538 e538: 12 total, 11 up, 12 in
>2018-07-11 10:31:03.050487 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e538: 12 total, 11 up, 12 in
>2018-07-11 10:31:03.183506 7f29149e4700  1 mon.magna102@0(leader).osd e539 e539: 12 total, 11 up, 12 in
>2018-07-11 10:31:03.217348 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e539: 12 total, 11 up, 12 in
>2018-07-11 10:31:03.950683 7f29149e4700  1 mon.magna102@0(leader).log v20735 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20735)
>2018-07-11 10:31:04.250896 7f29149e4700  1 mon.magna102@0(leader).osd e540 e540: 12 total, 11 up, 12 in
>2018-07-11 10:31:04.284642 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e540: 12 total, 11 up, 12 in
>2018-07-11 10:31:04.351331 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10508. Immutable memtables: 0.
>
>2018-07-11 10:31:04.351373 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 921] Syncing log #10505
>2018-07-11 10:31:04.367961 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.351364) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:31:04.367972 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 921] Flushing memtable with next log file: 10508
>
>2018-07-11 10:31:04.367999 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064367991, "job": 921, "event": "flush_started", "num_memtables": 1, "num_entries": 1315, "num_deletes": 251, "memory_usage": 2414120}
>2018-07-11 10:31:04.368002 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 921] Level-0 flush table #10509: started
>2018-07-11 10:31:04.418511 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064418496, "cf_name": "default", "job": 921, "event": "table_file_creation", "file_number": 10509, "file_size": 2298608, "table_properties": {"data_size": 2280051, "index_size": 9817, "filter_size": 7816, "raw_key_size": 11985, "raw_average_key_size": 20, "raw_value_size": 2264873, "raw_average_value_size": 3891, "num_data_blocks": 298, "num_entries": 582, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:31:04.418521 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 921] Level-0 flush table #10509: 2298608 bytes OK
>2018-07-11 10:31:04.468537 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.435028) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10509 started
>2018-07-11 10:31:04.468546 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.468499) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10509: memtable #1 done
>2018-07-11 10:31:04.468548 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.468514) EVENT_LOG_v1 {"time_micros": 1531305064468508, "job": 921, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:31:04.468550 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.468527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:31:04.468570 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 921] Try to delete WAL files size 2387662, prev total WAL file size 2387662, number of live WAL files 2.
>
>2018-07-11 10:31:04.469177 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:31:04.469252 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.469245) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:31:04.469261 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 922] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:31:04.469268 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 921 Base level 0, inputs: [10509(2244KB)], [10507(15MB)]
>
>2018-07-11 10:31:04.469282 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064469272, "job": 922, "event": "compaction_started", "files_L0": [10509], "files_L1": [10507], "score": -1, "input_data_size": 18770455}
>2018-07-11 10:31:04.599936 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.1 10.8.128.106:6804/12677 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.599960 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.1 10.8.128.106:6804/12677 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600024 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.5 10.8.128.113:6804/15263 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600036 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.5 10.8.128.113:6804/15263 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600142 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.3 10.8.128.106:6801/12650 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600154 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.106:6801/12650 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600236 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.8 10.8.128.108:6800/11967 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600247 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.8 10.8.128.108:6800/11967 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600320 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.0 10.8.128.113:6800/15255 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600331 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.113:6800/15255 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600401 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.6 10.8.128.115:6800/13323 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600412 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.6 10.8.128.115:6800/13323 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600487 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.11 10.8.128.114:6800/15919 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600499 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.11 10.8.128.114:6800/15919 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600567 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.2 10.8.128.106:6800/12651 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600579 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.2 10.8.128.106:6800/12651 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600649 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.7 10.8.128.114:6804/15885 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600660 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.7 10.8.128.114:6804/15885 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.600731 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.4 10.8.128.113:6808/15287 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:04.600743 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.4 10.8.128.113:6808/15287 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:04.660775 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 922] Generated table #10510: 3507 keys, 16732381 bytes
>2018-07-11 10:31:04.660804 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064660789, "cf_name": "default", "job": 922, "event": "table_file_creation", "file_number": 10510, "file_size": 16732381, "table_properties": {"data_size": 16605862, "index_size": 71887, "filter_size": 53705, "raw_key_size": 70871, "raw_average_key_size": 20, "raw_value_size": 16505545, "raw_average_value_size": 4706, "num_data_blocks": 2092, "num_entries": 3507, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:31:04.694103 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 922] Compacted 1@0 + 1@1 files to L1 => 16732381 bytes
>2018-07-11 10:31:04.727615 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.727546) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.5 rd, 74.4 wr, level 1, files in(1, 1) out(1) MB in(2.2, 15.7) out(16.0), read-write-amplify(15.4) write-amplify(7.3) OK, records in: 4021, records dropped: 514
>
>2018-07-11 10:31:04.727624 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:31:04.727567) EVENT_LOG_v1 {"time_micros": 1531305064727557, "job": 922, "event": "compaction_finished", "compaction_time_micros": 224804, "output_level": 1, "num_output_files": 1, "total_output_size": 16732381, "num_input_records": 4021, "num_output_records": 3507, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:31:04.728157 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064728153, "job": 922, "event": "table_file_deletion", "file_number": 10509}
>2018-07-11 10:31:04.731408 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305064731405, "job": 922, "event": "table_file_deletion", "file_number": 10507}
>2018-07-11 10:31:05.319958 7f29149e4700  1 mon.magna102@0(leader).log v20736 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20736)
>2018-07-11 10:31:05.351157 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 1 pg inactive, 16 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:31:05.351206 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 214713/24217500 objects degraded (0.887%), 7 pgs degraded (PG_DEGRADED)
>2018-07-11 10:31:06.395251 7f29149e4700  1 mon.magna102@0(leader).log v20737 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20737)
>2018-07-11 10:31:07.395250 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 10:31:07.570607 7f29149e4700  1 mon.magna102@0(leader).log v20738 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20738)
>2018-07-11 10:31:10.896204 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:31:11.046126 7f29149e4700  1 mon.magna102@0(leader).log v20739 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20739)
>2018-07-11 10:31:12.983305 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 22 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:31:12.983335 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 448216/24217500 objects degraded (1.851%), 19 pgs degraded (PG_DEGRADED)
>2018-07-11 10:31:14.021446 7f29149e4700  1 mon.magna102@0(leader).log v20740 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20740)
>2018-07-11 10:31:15.089513 7f29149e4700  1 mon.magna102@0(leader).log v20741 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20741)
>2018-07-11 10:31:20.108886 7f29191ed700  1 mon.magna102@0(leader).osd e540 prepare_failure osd.9 10.8.128.107:6804/12617 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:31:20.108900 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.9 10.8.128.107:6804/12617 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:31:20.231802 7f29149e4700  1 mon.magna102@0(leader).log v20742 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20742)
>2018-07-11 10:31:21.231859 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:31:21.282633 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 10:31:21.382351 7f29149e4700  1 mon.magna102@0(leader).osd e541 e541: 12 total, 12 up, 12 in
>2018-07-11 10:31:21.415956 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14975 boot
>2018-07-11 10:31:21.416018 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e541: 12 total, 12 up, 12 in
>2018-07-11 10:31:21.482552 7f29149e4700  1 mon.magna102@0(leader).log v20743 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20743)
>2018-07-11 10:31:21.566166 7f29149e4700  1 mon.magna102@0(leader).osd e542 e542: 12 total, 12 up, 12 in
>2018-07-11 10:31:21.600060 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e542: 12 total, 12 up, 12 in
>2018-07-11 10:31:22.558118 7f29149e4700  1 mon.magna102@0(leader).log v20744 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20744)
>2018-07-11 10:31:23.558110 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 447946/24217500 objects degraded (1.850%), 18 pgs degraded (PG_DEGRADED)
>2018-07-11 10:31:23.716857 7f29149e4700  1 mon.magna102@0(leader).log v20745 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20745)
>2018-07-11 10:31:24.903518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 32 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:31:25.050659 7f29149e4700  1 mon.magna102@0(leader).log v20746 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20746)
>2018-07-11 10:31:26.904435 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 0,1,8 (REQUEST_SLOW)
>2018-07-11 10:31:26.904480 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60262/24217500 objects degraded (0.249%), 5 pgs degraded)
>2018-07-11 10:31:27.059348 7f29149e4700  1 mon.magna102@0(leader).log v20747 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20747)
>2018-07-11 10:31:40.911623 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 53 slow requests are blocked > 32 sec. Implicated osds 0,1,8,10 (REQUEST_SLOW)
>2018-07-11 10:31:41.068812 7f29149e4700  1 mon.magna102@0(leader).log v20748 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20748)
>2018-07-11 10:31:52.457898 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 10:31:52.457912 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 10:31:52.577903 7f29149e4700  1 mon.magna102@0(leader).log v20749 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20749)
>2018-07-11 10:31:53.755344 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 10:31:53.755358 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 10:31:54.711587 7f29149e4700  1 mon.magna102@0(leader).log v20750 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20750)
>2018-07-11 10:31:54.857841 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 10:31:54.857855 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 10:31:55.273533 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 10:31:55.273547 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 10:31:55.602336 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 10:31:55.602350 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 10:31:55.778443 7f29149e4700  1 mon.magna102@0(leader).log v20751 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20751)
>2018-07-11 10:31:55.811921 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 10:31:55.811935 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 10:31:56.472247 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 10:31:56.472260 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 10:31:56.717529 7f29191ed700  1 mon.magna102@0(leader).osd e542 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 10:31:56.717542 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 10:31:56.845524 7f29149e4700  1 mon.magna102@0(leader).log v20752 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20752)
>2018-07-11 10:32:02.909886 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:32:03.005410 7f291b9f2700  1 mon.magna102@0(leader).osd e542  we have enough reporters to mark osd.10 down
>2018-07-11 10:32:03.005430 7f291b9f2700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (5 reporters from different host after 26.287954 >= grace 22.504638)
>2018-07-11 10:32:03.006276 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 10:32:03.131222 7f29149e4700  1 mon.magna102@0(leader).osd e543 e543: 12 total, 11 up, 12 in
>2018-07-11 10:32:03.173326 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e543: 12 total, 11 up, 12 in
>2018-07-11 10:32:03.256394 7f29149e4700  1 mon.magna102@0(leader).log v20753 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20753)
>2018-07-11 10:32:03.348366 7f29149e4700  1 mon.magna102@0(leader).osd e544 e544: 12 total, 11 up, 12 in
>2018-07-11 10:32:03.390572 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e544: 12 total, 11 up, 12 in
>2018-07-11 10:32:04.356913 7f29149e4700  1 mon.magna102@0(leader).log v20754 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20754)
>2018-07-11 10:32:05.356948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 8 pgs inactive, 30 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:05.356994 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 52024/24217500 objects degraded (0.215%), 2 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:05.548995 7f29149e4700  1 mon.magna102@0(leader).log v20755 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20755)
>2018-07-11 10:32:13.006700 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 23 pgs inactive, 29 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:13.006744 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 448216/24217500 objects degraded (1.851%), 19 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:14.058069 7f29149e4700  1 mon.magna102@0(leader).log v20756 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20756)
>2018-07-11 10:32:18.716853 7f29149e4700  1 mon.magna102@0(leader).osd e545 e545: 12 total, 11 up, 12 in
>2018-07-11 10:32:18.750568 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e545: 12 total, 11 up, 12 in
>2018-07-11 10:32:19.716841 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 23 pgs inactive, 28 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:19.716888 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 508273/24217500 objects degraded (2.099%), 19 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:19.716906 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 0,1,8 (REQUEST_SLOW)
>2018-07-11 10:32:19.850680 7f29149e4700  1 mon.magna102@0(leader).osd e546 e546: 12 total, 11 up, 12 in
>2018-07-11 10:32:19.884080 7f29149e4700  1 mon.magna102@0(leader).log v20757 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20757)
>2018-07-11 10:32:19.884479 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e546: 12 total, 11 up, 12 in
>2018-07-11 10:32:20.917667 7f29149e4700  1 mon.magna102@0(leader).log v20758 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20758)
>2018-07-11 10:32:20.934311 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 10:32:21.984661 7f29149e4700  1 mon.magna102@0(leader).log v20759 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20759)
>2018-07-11 10:32:22.868207 7f29149e4700  1 mon.magna102@0(leader).osd e547 e547: 12 total, 11 up, 12 in
>2018-07-11 10:32:22.901917 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e547: 12 total, 11 up, 12 in
>2018-07-11 10:32:23.993737 7f29149e4700  1 mon.magna102@0(leader).log v20760 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20760)
>2018-07-11 10:32:24.993791 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:24.993838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1440427/24217500 objects degraded (5.948%), 45 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:24.993857 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:32:25.144171 7f29149e4700  1 mon.magna102@0(leader).log v20761 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20761)
>2018-07-11 10:32:26.498023 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 10:32:26.561077 7f29149e4700  1 mon.magna102@0(leader).osd e548 e548: 12 total, 12 up, 12 in
>2018-07-11 10:32:26.594691 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14975 boot
>2018-07-11 10:32:26.594768 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e548: 12 total, 12 up, 12 in
>2018-07-11 10:32:26.661372 7f29149e4700  1 mon.magna102@0(leader).log v20762 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20762)
>2018-07-11 10:32:26.744898 7f29149e4700  1 mon.magna102@0(leader).osd e549 e549: 12 total, 12 up, 12 in
>2018-07-11 10:32:26.778847 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e549: 12 total, 12 up, 12 in
>2018-07-11 10:32:27.728497 7f29149e4700  1 mon.magna102@0(leader).log v20763 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20763)
>2018-07-11 10:32:30.937473 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs inactive, 21 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:30.937520 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 416090/24217500 objects degraded (1.718%), 11 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:30.937538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 71 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:32:30.937567 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 10:32:31.088816 7f29149e4700  1 mon.magna102@0(leader).log v20764 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 20764)
>2018-07-11 10:32:34.705825 7f29149e4700  1 mon.magna102@0(leader).log v20765 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20765)
>2018-07-11 10:32:38.007690 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 13 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:32:38.007721 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210858/24217500 objects degraded (0.871%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:38.148107 7f29149e4700  1 mon.magna102@0(leader).log v20766 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20766)
>2018-07-11 10:32:40.942647 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 110 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:32:41.090282 7f29149e4700  1 mon.magna102@0(leader).log v20767 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20767)
>2018-07-11 10:32:43.007938 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210857/24217500 objects degraded (0.871%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:43.123938 7f29149e4700  1 mon.magna102@0(leader).log v20768 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20768)
>2018-07-11 10:32:46.945576 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 116 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:32:47.099398 7f29149e4700  1 mon.magna102@0(leader).log v20769 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20769)
>2018-07-11 10:32:48.083475 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10511. Immutable memtables: 0.
>
>2018-07-11 10:32:48.083518 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 923] Syncing log #10508
>2018-07-11 10:32:48.099686 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.083509) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:32:48.099697 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 923] Flushing memtable with next log file: 10511
>
>2018-07-11 10:32:48.099724 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168099716, "job": 923, "event": "flush_started", "num_memtables": 1, "num_entries": 1229, "num_deletes": 500, "memory_usage": 2052688}
>2018-07-11 10:32:48.099727 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 923] Level-0 flush table #10512: started
>2018-07-11 10:32:48.158549 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168158535, "cf_name": "default", "job": 923, "event": "table_file_creation", "file_number": 10512, "file_size": 1817101, "table_properties": {"data_size": 1803004, "index_size": 6552, "filter_size": 6625, "raw_key_size": 15394, "raw_average_key_size": 20, "raw_value_size": 1788143, "raw_average_value_size": 2396, "num_data_blocks": 199, "num_entries": 746, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 10:32:48.158559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 923] Level-0 flush table #10512: 1817101 bytes OK
>2018-07-11 10:32:48.208520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.175092) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10512 started
>2018-07-11 10:32:48.208528 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.208470) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10512: memtable #1 done
>2018-07-11 10:32:48.208531 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.208485) EVENT_LOG_v1 {"time_micros": 1531305168208479, "job": 923, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:32:48.208533 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.208510) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:32:48.208539 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 923] Try to delete WAL files size 2024947, prev total WAL file size 2024947, number of live WAL files 2.
>
>2018-07-11 10:32:48.209096 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:32:48.209158 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.209151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 10:32:48.209168 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 924] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:32:48.209174 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 923 Base level 0, inputs: [10512(1774KB)], [10510(15MB)]
>
>2018-07-11 10:32:48.209198 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168209177, "job": 924, "event": "compaction_started", "files_L0": [10512], "files_L1": [10510], "score": -1, "input_data_size": 18549482}
>2018-07-11 10:32:48.400958 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 924] Generated table #10513: 3240 keys, 16063813 bytes
>2018-07-11 10:32:48.401002 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168400975, "cf_name": "default", "job": 924, "event": "table_file_creation", "file_number": 10513, "file_size": 16063813, "table_properties": {"data_size": 15942502, "index_size": 69054, "filter_size": 51330, "raw_key_size": 65428, "raw_average_key_size": 20, "raw_value_size": 15847998, "raw_average_value_size": 4891, "num_data_blocks": 2021, "num_entries": 3240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:32:48.442486 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 924] Compacted 1@0 + 1@1 files to L1 => 16063813 bytes
>2018-07-11 10:32:48.484349 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.484276) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(1.7, 16.0) out(15.3), read-write-amplify(19.0) write-amplify(8.8) OK, records in: 4253, records dropped: 1013
>
>2018-07-11 10:32:48.484358 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:32:48.484298) EVENT_LOG_v1 {"time_micros": 1531305168484288, "job": 924, "event": "compaction_finished", "compaction_time_micros": 233273, "output_level": 1, "num_output_files": 1, "total_output_size": 16063813, "num_input_records": 4253, "num_output_records": 3240, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:32:48.484802 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168484798, "job": 924, "event": "table_file_deletion", "file_number": 10512}
>2018-07-11 10:32:48.487800 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305168487797, "job": 924, "event": "table_file_deletion", "file_number": 10510}
>2018-07-11 10:32:48.487916 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:32:53.049934 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 124 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:32:54.077085 7f29149e4700  1 mon.magna102@0(leader).log v20770 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20770)
>2018-07-11 10:32:55.077059 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210856/24217500 objects degraded (0.871%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:32:55.227433 7f29149e4700  1 mon.magna102@0(leader).log v20771 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20771)
>2018-07-11 10:32:58.050227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 132 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:32:58.161095 7f29149e4700  1 mon.magna102@0(leader).log v20772 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20772)
>2018-07-11 10:33:02.910062 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:33:04.955538 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210855/24217500 objects degraded (0.871%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:33:05.103520 7f29149e4700  1 mon.magna102@0(leader).log v20773 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20773)
>2018-07-11 10:33:06.956274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 136 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:07.105098 7f29149e4700  1 mon.magna102@0(leader).log v20774 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20774)
>2018-07-11 10:33:13.050834 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 146 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:14.097441 7f29149e4700  1 mon.magna102@0(leader).log v20775 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20775)
>2018-07-11 10:33:15.097460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210854/24217500 objects degraded (0.871%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:33:15.247667 7f29149e4700  1 mon.magna102@0(leader).log v20776 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20776)
>2018-07-11 10:33:18.051072 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 148 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:18.164764 7f29149e4700  1 mon.magna102@0(leader).log v20777 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20777)
>2018-07-11 10:33:20.963800 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 14 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:33:21.115284 7f29149e4700  1 mon.magna102@0(leader).log v20778 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20778)
>2018-07-11 10:33:22.964720 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210854/24217500 objects degraded (0.871%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED)
>2018-07-11 10:33:23.082044 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 152 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:23.115622 7f29149e4700  1 mon.magna102@0(leader).log v20779 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20779)
>2018-07-11 10:33:24.182474 7f29149e4700  1 mon.magna102@0(leader).log v20780 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20780)
>2018-07-11 10:33:26.967012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 24 pgs inactive, 38 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:33:27.116356 7f29149e4700  1 mon.magna102@0(leader).log v20781 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20781)
>2018-07-11 10:33:28.082276 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210853/24217500 objects degraded (0.871%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED)
>2018-07-11 10:33:28.082322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 154 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:28.183192 7f29149e4700  1 mon.magna102@0(leader).log v20782 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20782)
>2018-07-11 10:33:33.082541 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 38 pgs inactive, 38 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:33:33.082585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 156 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:34.125449 7f29149e4700  1 mon.magna102@0(leader).log v20783 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20783)
>2018-07-11 10:33:35.125452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210852/24217500 objects degraded (0.871%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED)
>2018-07-11 10:33:35.275823 7f29149e4700  1 mon.magna102@0(leader).log v20784 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20784)
>2018-07-11 10:33:38.082803 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 160 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:38.201160 7f29149e4700  1 mon.magna102@0(leader).log v20785 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20785)
>2018-07-11 10:33:44.978028 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 210851/24217500 objects degraded (0.871%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED)
>2018-07-11 10:33:45.135330 7f29149e4700  1 mon.magna102@0(leader).log v20786 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20786)
>2018-07-11 10:33:46.979040 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 162 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:47.127330 7f29149e4700  1 mon.magna102@0(leader).log v20787 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20787)
>2018-07-11 10:33:52.944503 7f29149e4700  1 mon.magna102@0(leader).osd e550 e550: 12 total, 12 up, 12 in
>2018-07-11 10:33:52.978230 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e550: 12 total, 12 up, 12 in
>2018-07-11 10:33:53.083413 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 174 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:54.012877 7f29149e4700  1 mon.magna102@0(leader).log v20788 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20788)
>2018-07-11 10:33:54.079799 7f29149e4700  1 mon.magna102@0(leader).osd e551 e551: 12 total, 12 up, 12 in
>2018-07-11 10:33:54.113520 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e551: 12 total, 12 up, 12 in
>2018-07-11 10:33:55.079782 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 24 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:33:55.079826 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 136378/24217500 objects degraded (0.563%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED)
>2018-07-11 10:33:55.213608 7f29149e4700  1 mon.magna102@0(leader).osd e552 e552: 12 total, 12 up, 12 in
>2018-07-11 10:33:55.247015 7f29149e4700  1 mon.magna102@0(leader).log v20789 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20789)
>2018-07-11 10:33:55.247369 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e552: 12 total, 12 up, 12 in
>2018-07-11 10:33:56.238932 7f29149e4700  1 mon.magna102@0(leader).osd e553 e553: 12 total, 12 up, 12 in
>2018-07-11 10:33:56.272618 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e553: 12 total, 12 up, 12 in
>2018-07-11 10:33:56.347591 7f29149e4700  1 mon.magna102@0(leader).log v20790 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20790)
>2018-07-11 10:33:57.306118 7f29149e4700  1 mon.magna102@0(leader).osd e554 e554: 12 total, 12 up, 12 in
>2018-07-11 10:33:57.348048 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e554: 12 total, 12 up, 12 in
>2018-07-11 10:33:57.389505 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 10:33:57.423103 7f29149e4700  1 mon.magna102@0(leader).log v20791 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20791)
>2018-07-11 10:33:58.083664 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 176 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:33:58.506878 7f29149e4700  1 mon.magna102@0(leader).log v20792 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20792)
>2018-07-11 10:34:00.986419 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs inactive, 10 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:34:01.157352 7f29149e4700  1 mon.magna102@0(leader).log v20793 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20793)
>2018-07-11 10:34:02.910253 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:34:03.083902 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 17/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:03.083933 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 111 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:34:04.149344 7f29149e4700  1 mon.magna102@0(leader).log v20794 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20794)
>2018-07-11 10:34:05.149294 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 111 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:34:05.324711 7f29149e4700  1 mon.magna102@0(leader).log v20795 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20795)
>2018-07-11 10:34:08.084164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 15/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:08.216695 7f29149e4700  1 mon.magna102@0(leader).log v20796 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20796)
>2018-07-11 10:34:13.084413 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 14/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:14.158997 7f29149e4700  1 mon.magna102@0(leader).log v20797 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20797)
>2018-07-11 10:34:15.158974 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:34:15.334351 7f29149e4700  1 mon.magna102@0(leader).log v20798 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20798)
>2018-07-11 10:34:20.995804 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 13/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:20.995849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:34:21.168396 7f29149e4700  1 mon.magna102@0(leader).log v20799 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20799)
>2018-07-11 10:34:26.998522 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:34:27.168961 7f29149e4700  1 mon.magna102@0(leader).log v20800 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20800)
>2018-07-11 10:34:28.085011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 11/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:28.269311 7f29149e4700  1 mon.magna102@0(leader).log v20801 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20801)
>2018-07-11 10:34:35.002571 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 9/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:35.154620 7f29149e4700  1 mon.magna102@0(leader).log v20802 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20802)
>2018-07-11 10:34:41.005277 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 8/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:41.155200 7f29149e4700  1 mon.magna102@0(leader).log v20803 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20803)
>2018-07-11 10:34:51.010169 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 5/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:34:51.164492 7f29149e4700  1 mon.magna102@0(leader).log v20804 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20804)
>2018-07-11 10:35:02.910436 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:35:05.016953 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 3/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:35:05.174055 7f29149e4700  1 mon.magna102@0(leader).log v20805 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20805)
>2018-07-11 10:35:11.019626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 2/24217500 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED)
>2018-07-11 10:35:11.174619 7f29149e4700  1 mon.magna102@0(leader).log v20806 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20806)
>2018-07-11 10:35:18.158739 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10514. Immutable memtables: 0.
>
>2018-07-11 10:35:18.158786 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 925] Syncing log #10511
>2018-07-11 10:35:18.175286 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.158777) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:35:18.175298 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 925] Flushing memtable with next log file: 10514
>
>2018-07-11 10:35:18.175325 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318175316, "job": 925, "event": "flush_started", "num_memtables": 1, "num_entries": 1122, "num_deletes": 250, "memory_usage": 2347496}
>2018-07-11 10:35:18.175328 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 925] Level-0 flush table #10515: started
>2018-07-11 10:35:18.225820 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318225806, "cf_name": "default", "job": 925, "event": "table_file_creation", "file_number": 10515, "file_size": 1627077, "table_properties": {"data_size": 1615266, "index_size": 5494, "filter_size": 5397, "raw_key_size": 10011, "raw_average_key_size": 21, "raw_value_size": 1604807, "raw_average_value_size": 3421, "num_data_blocks": 169, "num_entries": 469, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:35:18.225830 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 925] Level-0 flush table #10515: 1627077 bytes OK
>2018-07-11 10:35:18.291034 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.242363) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10515 started
>2018-07-11 10:35:18.291042 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.290998) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10515: memtable #1 done
>2018-07-11 10:35:18.291044 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.291012) EVENT_LOG_v1 {"time_micros": 1531305318291006, "job": 925, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:35:18.291046 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.291024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:35:18.291065 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 925] Try to delete WAL files size 2324691, prev total WAL file size 2324691, number of live WAL files 2.
>
>2018-07-11 10:35:18.291686 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:35:18.291879 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 10:35:18.291885 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 67345.1 total, 675.2 interval
>Cumulative writes: 91K writes, 431K keys, 91K commit groups, 1.0 writes per commit group, ingest: 0.72 GB, 0.01 MB/s
>Cumulative WAL: 91K writes, 91K syncs, 1.00 writes per sync, written: 0.72 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 943 writes, 4650 keys, 943 commit groups, 1.0 writes per commit group, ingest: 8.01 MB, 0.01 MB/s
>Interval WAL: 943 writes, 943 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.55 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        37       463    0.081       0      0
>  L1      1/1   15.32 MB   0.0      7.3     0.6      6.7       6.7     -0.0       0.0  10.7     72.6     66.4       103       461    0.223   1648K   197K
> Sum      2/2   16.87 MB   0.0      7.3     0.6      6.7       7.3      0.6       0.0  11.7     53.2     53.2       140       924    0.152   1648K   197K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.7     59.2     58.5         1         8    0.148     16K   2466
>Uptime(secs): 67345.1 total, 67345.1 interval
>Flush(GB): cumulative 0.623, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.28 GB write, 0.11 MB/s write, 7.28 GB read, 0.11 MB/s read, 140.2 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.55 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        37       463    0.081       0      0
>  L1      1/1   15.32 MB   0.0      7.3     0.6      6.7       6.7     -0.0       0.0  10.7     72.6     66.4       103       461    0.223   1648K   197K
> Sum      2/2   16.87 MB   0.0      7.3     0.6      6.7       7.3      0.6       0.0  11.7     53.2     53.2       140       924    0.152   1648K   197K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 67345.1 total, 67345.1 interval
>Flush(GB): cumulative 0.623, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.28 GB write, 0.11 MB/s write, 7.28 GB read, 0.11 MB/s read, 140.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 10:35:18.291906 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.291900) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:35:18.291911 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 926] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:35:18.291916 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 925 Base level 0, inputs: [10515(1588KB)], [10513(15MB)]
>
>2018-07-11 10:35:18.291928 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318291919, "job": 926, "event": "compaction_started", "files_L0": [10515], "files_L1": [10513], "score": -1, "input_data_size": 17690890}
>2018-07-11 10:35:18.476567 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 926] Generated table #10516: 3266 keys, 16439814 bytes
>2018-07-11 10:35:18.476611 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318476584, "cf_name": "default", "job": 926, "event": "table_file_creation", "file_number": 10516, "file_size": 16439814, "table_properties": {"data_size": 16318452, "index_size": 68404, "filter_size": 52031, "raw_key_size": 65640, "raw_average_key_size": 20, "raw_value_size": 16224112, "raw_average_value_size": 4967, "num_data_blocks": 2008, "num_entries": 3266, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:35:18.509744 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 926] Compacted 1@0 + 1@1 files to L1 => 16439814 bytes
>2018-07-11 10:35:18.544513 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.544446) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.2 rd, 75.5 wr, level 1, files in(1, 1) out(1) MB in(1.6, 15.3) out(15.7), read-write-amplify(21.0) write-amplify(10.1) OK, records in: 3709, records dropped: 443
>
>2018-07-11 10:35:18.544522 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:35:18.544468) EVENT_LOG_v1 {"time_micros": 1531305318544458, "job": 926, "event": "compaction_finished", "compaction_time_micros": 217799, "output_level": 1, "num_output_files": 1, "total_output_size": 16439814, "num_input_records": 3709, "num_output_records": 3266, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:35:18.544927 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318544924, "job": 926, "event": "table_file_deletion", "file_number": 10515}
>2018-07-11 10:35:18.548087 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305318548085, "job": 926, "event": "table_file_deletion", "file_number": 10513}
>2018-07-11 10:35:18.548207 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:35:21.024599 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/24217500 objects degraded (0.000%), 1 pg degraded)
>2018-07-11 10:35:21.295451 7f29149e4700  1 mon.magna102@0(leader).log v20807 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20807)
>2018-07-11 10:35:24.712390 7f29149e4700  1 mon.magna102@0(leader).log v20808 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20808)
>2018-07-11 10:35:31.029569 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:35:31.179832 7f29149e4700  1 mon.magna102@0(leader).log v20809 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20809)
>2018-07-11 10:35:35.031415 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 10 pgs peering)
>2018-07-11 10:35:35.031461 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:35:35.197008 7f29149e4700  1 mon.magna102@0(leader).log v20810 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20810)
>2018-07-11 10:35:49.037297 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:35:49.189914 7f29149e4700  1 mon.magna102@0(leader).log v20811 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20811)
>2018-07-11 10:35:58.126600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:35:58.248921 7f29149e4700  1 mon.magna102@0(leader).log v20812 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20812)
>2018-07-11 10:36:02.910619 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:36:20.888707 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:36:20.888730 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/468742182' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:36:21.018535 7f29149e4700  1 mon.magna102@0(leader).log v20813 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20813)
>2018-07-11 10:36:21.102308 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10517. Immutable memtables: 0.
>
>2018-07-11 10:36:21.135607 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.102370) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:36:21.135621 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 927] Flushing memtable with next log file: 10517
>
>2018-07-11 10:36:21.135635 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381135627, "job": 927, "event": "flush_started", "num_memtables": 1, "num_entries": 529, "num_deletes": 251, "memory_usage": 655352}
>2018-07-11 10:36:21.135639 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 927] Level-0 flush table #10518: started
>2018-07-11 10:36:21.211256 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381211242, "cf_name": "default", "job": 927, "event": "table_file_creation", "file_number": 10518, "file_size": 616266, "table_properties": {"data_size": 610255, "index_size": 2551, "filter_size": 2541, "raw_key_size": 6976, "raw_average_key_size": 20, "raw_value_size": 603750, "raw_average_value_size": 1760, "num_data_blocks": 78, "num_entries": 343, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:36:21.211267 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 927] Level-0 flush table #10518: 616266 bytes OK
>2018-07-11 10:36:21.261272 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.227798) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10518 started
>2018-07-11 10:36:21.261280 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.261236) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10518: memtable #1 done
>2018-07-11 10:36:21.261283 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.261251) EVENT_LOG_v1 {"time_micros": 1531305381261244, "job": 927, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:36:21.261285 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.261262) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:36:21.261291 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 927] Try to delete WAL files size 642947, prev total WAL file size 658206, number of live WAL files 2.
>
>2018-07-11 10:36:21.261518 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:36:21.261571 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.261564) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:36:21.261580 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 928] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:36:21.261587 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 927 Base level 0, inputs: [10518(601KB)], [10516(15MB)]
>
>2018-07-11 10:36:21.261599 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381261590, "job": 928, "event": "compaction_started", "files_L0": [10518], "files_L1": [10516], "score": -1, "input_data_size": 17056080}
>2018-07-11 10:36:21.436968 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 928] Generated table #10519: 3099 keys, 15000561 bytes
>2018-07-11 10:36:21.437017 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381436989, "cf_name": "default", "job": 928, "event": "table_file_creation", "file_number": 10519, "file_size": 15000561, "table_properties": {"data_size": 14889176, "index_size": 62765, "filter_size": 47693, "raw_key_size": 62369, "raw_average_key_size": 20, "raw_value_size": 14801110, "raw_average_value_size": 4776, "num_data_blocks": 1834, "num_entries": 3099, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:36:21.470168 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 928] Compacted 1@0 + 1@1 files to L1 => 15000561 bytes
>2018-07-11 10:36:21.503667 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.503600) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.8 rd, 71.9 wr, level 1, files in(1, 1) out(1) MB in(0.6, 15.7) out(14.3), read-write-amplify(52.0) write-amplify(24.3) OK, records in: 3609, records dropped: 510
>
>2018-07-11 10:36:21.503675 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:36:21.503622) EVENT_LOG_v1 {"time_micros": 1531305381503612, "job": 928, "event": "compaction_finished", "compaction_time_micros": 208552, "output_level": 1, "num_output_files": 1, "total_output_size": 15000561, "num_input_records": 3609, "num_output_records": 3099, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:36:21.503844 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381503840, "job": 928, "event": "table_file_deletion", "file_number": 10518}
>2018-07-11 10:36:21.506745 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305381506742, "job": 928, "event": "table_file_deletion", "file_number": 10516}
>2018-07-11 10:37:01.069422 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:37:01.240502 7f29149e4700  1 mon.magna102@0(leader).log v20814 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20814)
>2018-07-11 10:37:02.910802 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:37:09.072851 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:37:09.234131 7f29149e4700  1 mon.magna102@0(leader).log v20815 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20815)
>2018-07-11 10:37:25.079863 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:37:25.227050 7f29149e4700  1 mon.magna102@0(leader).log v20816 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20816)
>2018-07-11 10:37:33.130105 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:37:33.244476 7f29149e4700  1 mon.magna102@0(leader).log v20817 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20817)
>2018-07-11 10:38:02.910978 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:38:39.112395 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 10:38:39.283824 7f29149e4700  1 mon.magna102@0(leader).log v20818 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20818)
>2018-07-11 10:38:41.113334 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 10:38:41.284133 7f29149e4700  1 mon.magna102@0(leader).log v20819 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20819)
>2018-07-11 10:38:59.121121 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:38:59.293996 7f29149e4700  1 mon.magna102@0(leader).log v20820 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20820)
>2018-07-11 10:39:02.911152 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:39:08.160236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:39:08.253051 7f29149e4700  1 mon.magna102@0(leader).log v20821 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20821)
>2018-07-11 10:39:12.010370 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:39:12.010391 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/62442639' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:39:12.253491 7f29149e4700  1 mon.magna102@0(leader).log v20822 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20822)
>2018-07-11 10:40:02.911357 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:40:11.153441 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:40:11.309630 7f29149e4700  1 mon.magna102@0(leader).log v20823 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20823)
>2018-07-11 10:40:15.155344 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:40:15.335085 7f29149e4700  1 mon.magna102@0(leader).log v20824 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20824)
>2018-07-11 10:40:31.162155 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 10:40:31.320906 7f29149e4700  1 mon.magna102@0(leader).log v20825 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20825)
>2018-07-11 10:40:37.164737 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:40:37.313405 7f29149e4700  1 mon.magna102@0(leader).log v20826 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20826)
>2018-07-11 10:41:02.911538 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:41:28.190360 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:41:28.190384 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1414229544' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:41:28.325245 7f29149e4700  1 mon.magna102@0(leader).log v20827 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20827)
>2018-07-11 10:41:47.195897 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:41:47.353086 7f29149e4700  1 mon.magna102@0(leader).log v20828 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20828)
>2018-07-11 10:41:49.196713 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:41:49.353390 7f29149e4700  1 mon.magna102@0(leader).log v20829 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20829)
>2018-07-11 10:42:02.911725 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:42:05.203775 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:42:05.371423 7f29149e4700  1 mon.magna102@0(leader).log v20830 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20830)
>2018-07-11 10:42:13.246806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:42:13.380501 7f29149e4700  1 mon.magna102@0(leader).log v20831 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20831)
>2018-07-11 10:42:21.210831 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:42:21.381324 7f29149e4700  1 mon.magna102@0(leader).log v20832 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20832)
>2018-07-11 10:43:02.911910 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:43:17.235595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:43:17.386509 7f29149e4700  1 mon.magna102@0(leader).log v20833 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20833)
>2018-07-11 10:43:25.239177 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:43:25.387223 7f29149e4700  1 mon.magna102@0(leader).log v20834 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20834)
>2018-07-11 10:43:38.355235 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10520. Immutable memtables: 0.
>
>2018-07-11 10:43:38.355281 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 929] Syncing log #10517
>2018-07-11 10:43:38.371431 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.355271) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:43:38.371444 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 929] Flushing memtable with next log file: 10520
>
>2018-07-11 10:43:38.371458 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818371450, "job": 929, "event": "flush_started", "num_memtables": 1, "num_entries": 1956, "num_deletes": 250, "memory_usage": 3802416}
>2018-07-11 10:43:38.371461 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 929] Level-0 flush table #10521: started
>2018-07-11 10:43:38.430480 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818430465, "cf_name": "default", "job": 929, "event": "table_file_creation", "file_number": 10521, "file_size": 2158830, "table_properties": {"data_size": 2141952, "index_size": 8668, "filter_size": 7286, "raw_key_size": 11522, "raw_average_key_size": 21, "raw_value_size": 2128156, "raw_average_value_size": 3890, "num_data_blocks": 270, "num_entries": 547, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:43:38.430490 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 929] Level-0 flush table #10521: 2158830 bytes OK
>2018-07-11 10:43:38.480493 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.447003) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10521 started
>2018-07-11 10:43:38.480501 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.480456) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10521: memtable #1 done
>2018-07-11 10:43:38.480505 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.480471) EVENT_LOG_v1 {"time_micros": 1531305818480465, "job": 929, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:43:38.480509 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.480483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:43:38.480515 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 929] Try to delete WAL files size 3765506, prev total WAL file size 3765506, number of live WAL files 2.
>
>2018-07-11 10:43:38.481465 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:43:38.481531 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.481512) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:43:38.481540 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 930] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:43:38.481546 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 929 Base level 0, inputs: [10521(2108KB)], [10519(14MB)]
>
>2018-07-11 10:43:38.481559 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818481550, "job": 930, "event": "compaction_started", "files_L0": [10521], "files_L1": [10519], "score": -1, "input_data_size": 17159391}
>2018-07-11 10:43:38.689579 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 930] Generated table #10522: 3351 keys, 16900104 bytes
>2018-07-11 10:43:38.689626 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818689597, "cf_name": "default", "job": 930, "event": "table_file_creation", "file_number": 10522, "file_size": 16900104, "table_properties": {"data_size": 16775272, "index_size": 70395, "filter_size": 53510, "raw_key_size": 67368, "raw_average_key_size": 20, "raw_value_size": 16678348, "raw_average_value_size": 4977, "num_data_blocks": 2066, "num_entries": 3351, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:43:38.731094 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 930] Compacted 1@0 + 1@1 files to L1 => 16900104 bytes
>2018-07-11 10:43:38.774144 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.774076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.8 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(2.1, 14.3) out(16.1), read-write-amplify(15.8) write-amplify(7.8) OK, records in: 3646, records dropped: 295
>
>2018-07-11 10:43:38.774153 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:38.774098) EVENT_LOG_v1 {"time_micros": 1531305818774088, "job": 930, "event": "compaction_finished", "compaction_time_micros": 249520, "output_level": 1, "num_output_files": 1, "total_output_size": 16900104, "num_input_records": 3646, "num_output_records": 3351, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:43:38.774631 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818774627, "job": 930, "event": "table_file_deletion", "file_number": 10521}
>2018-07-11 10:43:38.777612 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305818777609, "job": 930, "event": "table_file_deletion", "file_number": 10519}
>2018-07-11 10:43:38.777665 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:43:39.355086 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:43:39.524940 7f29149e4700  1 mon.magna102@0(leader).log v20835 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20835)
>2018-07-11 10:43:48.321841 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:43:48.442346 7f29149e4700  1 mon.magna102@0(leader).log v20836 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20836)
>2018-07-11 10:43:49.848918 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:43:49.848941 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1291297793' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:43:50.584371 7f29149e4700  1 mon.magna102@0(leader).log v20837 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20837)
>2018-07-11 10:43:50.651464 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10523. Immutable memtables: 0.
>
>2018-07-11 10:43:50.651509 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 931] Syncing log #10520
>2018-07-11 10:43:50.668019 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.651500) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:43:50.668030 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 931] Flushing memtable with next log file: 10523
>
>2018-07-11 10:43:50.668045 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305830668037, "job": 931, "event": "flush_started", "num_memtables": 1, "num_entries": 322, "num_deletes": 251, "memory_usage": 172728}
>2018-07-11 10:43:50.668048 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 931] Level-0 flush table #10524: started
>2018-07-11 10:43:50.710213 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305830710199, "cf_name": "default", "job": 931, "event": "table_file_creation", "file_number": 10524, "file_size": 162208, "table_properties": {"data_size": 159519, "index_size": 641, "filter_size": 1129, "raw_key_size": 5682, "raw_average_key_size": 20, "raw_value_size": 155271, "raw_average_value_size": 552, "num_data_blocks": 20, "num_entries": 281, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:43:50.710223 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 931] Level-0 flush table #10524: 162208 bytes OK
>2018-07-11 10:43:50.760248 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.726776) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10524 started
>2018-07-11 10:43:50.760257 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.760212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10524: memtable #1 done
>2018-07-11 10:43:50.760259 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.760226) EVENT_LOG_v1 {"time_micros": 1531305830760220, "job": 931, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:43:50.760262 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.760239) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:43:50.760269 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 931] Try to delete WAL files size 164179, prev total WAL file size 164179, number of live WAL files 2.
>
>2018-07-11 10:43:50.760396 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:43:50.760449 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:50.760441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:43:50.760458 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 932] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:43:50.760464 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 931 Base level 0, inputs: [10524(158KB)], [10522(16MB)]
>
>2018-07-11 10:43:50.760477 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305830760467, "job": 932, "event": "compaction_started", "files_L0": [10524], "files_L1": [10522], "score": -1, "input_data_size": 17062312}
>2018-07-11 10:43:50.944304 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 932] Generated table #10525: 3122 keys, 15010071 bytes
>2018-07-11 10:43:50.944342 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305830944326, "cf_name": "default", "job": 932, "event": "table_file_creation", "file_number": 10525, "file_size": 15010071, "table_properties": {"data_size": 14898592, "index_size": 62787, "filter_size": 47765, "raw_key_size": 62803, "raw_average_key_size": 20, "raw_value_size": 14810192, "raw_average_value_size": 4743, "num_data_blocks": 1834, "num_entries": 3122, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:43:50.977457 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 932] Compacted 1@0 + 1@1 files to L1 => 15010071 bytes
>2018-07-11 10:43:51.011007 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:51.010938) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.6 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.1) out(14.3), read-write-amplify(197.7) write-amplify(92.5) OK, records in: 3632, records dropped: 510
>
>2018-07-11 10:43:51.011016 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:43:51.010960) EVENT_LOG_v1 {"time_micros": 1531305831010951, "job": 932, "event": "compaction_finished", "compaction_time_micros": 216964, "output_level": 1, "num_output_files": 1, "total_output_size": 15010071, "num_input_records": 3632, "num_output_records": 3122, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:43:51.011115 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305831011111, "job": 932, "event": "table_file_deletion", "file_number": 10524}
>2018-07-11 10:43:51.014195 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531305831014192, "job": 932, "event": "table_file_deletion", "file_number": 10522}
>2018-07-11 10:44:02.912088 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:44:44.716710 7f29149e4700  1 mon.magna102@0(leader).log v20838 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20838)
>2018-07-11 10:44:54.725854 7f29149e4700  1 mon.magna102@0(leader).log v20839 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20839)
>2018-07-11 10:44:57.280838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:44:57.443020 7f29149e4700  1 mon.magna102@0(leader).log v20840 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20840)
>2018-07-11 10:44:59.281682 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:44:59.434994 7f29149e4700  1 mon.magna102@0(leader).log v20841 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20841)
>2018-07-11 10:45:02.912271 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:45:09.286067 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:45:09.435871 7f29149e4700  1 mon.magna102@0(leader).log v20842 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20842)
>2018-07-11 10:45:17.289607 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:45:17.436526 7f29149e4700  1 mon.magna102@0(leader).log v20843 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20843)
>2018-07-11 10:46:02.912412 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:46:27.320308 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:46:27.492531 7f29149e4700  1 mon.magna102@0(leader).log v20844 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20844)
>2018-07-11 10:46:29.321259 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:46:29.492828 7f29149e4700  1 mon.magna102@0(leader).log v20845 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20845)
>2018-07-11 10:46:47.328992 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 10:46:47.478748 7f29149e4700  1 mon.magna102@0(leader).log v20846 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20846)
>2018-07-11 10:46:53.370213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:46:53.487676 7f29149e4700  1 mon.magna102@0(leader).log v20847 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20847)
>2018-07-11 10:47:02.912583 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:47:50.949412 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.9 10.8.128.107:6804/12617 is reporting failure:1
>2018-07-11 10:47:50.949427 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.9 10.8.128.107:6804/12617
>2018-07-11 10:47:51.066493 7f29149e4700  1 mon.magna102@0(leader).log v20848 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20848)
>2018-07-11 10:47:51.696804 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.11 10.8.128.114:6800/15919 is reporting failure:1
>2018-07-11 10:47:51.696818 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.11 10.8.128.114:6800/15919
>2018-07-11 10:47:52.133323 7f29149e4700  1 mon.magna102@0(leader).log v20849 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20849)
>2018-07-11 10:47:53.401539 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.7 10.8.128.114:6804/15885 is reporting failure:1
>2018-07-11 10:47:53.401553 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.7 10.8.128.114:6804/15885
>2018-07-11 10:47:53.518536 7f29149e4700  1 mon.magna102@0(leader).log v20850 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20850)
>2018-07-11 10:47:53.836923 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.4 10.8.128.113:6808/15287 is reporting failure:1
>2018-07-11 10:47:53.836937 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.4 10.8.128.113:6808/15287
>2018-07-11 10:47:54.560207 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.1 10.8.128.106:6804/12677 is reporting failure:1
>2018-07-11 10:47:54.560223 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.1 10.8.128.106:6804/12677
>2018-07-11 10:47:54.593644 7f29149e4700  1 mon.magna102@0(leader).log v20851 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20851)
>2018-07-11 10:47:54.827307 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.0 10.8.128.113:6800/15255 is reporting failure:1
>2018-07-11 10:47:54.827321 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.0 10.8.128.113:6800/15255
>2018-07-11 10:47:55.438492 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.8 10.8.128.108:6800/11967 is reporting failure:1
>2018-07-11 10:47:55.438505 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.8 10.8.128.108:6800/11967
>2018-07-11 10:47:55.524233 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.5 10.8.128.113:6804/15263 is reporting failure:1
>2018-07-11 10:47:55.524246 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.5 10.8.128.113:6804/15263
>2018-07-11 10:47:55.542909 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.3 10.8.128.106:6801/12650 is reporting failure:1
>2018-07-11 10:47:55.542922 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.3 10.8.128.106:6801/12650
>2018-07-11 10:47:55.660530 7f29149e4700  1 mon.magna102@0(leader).log v20852 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20852)
>2018-07-11 10:47:55.817156 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.2 10.8.128.106:6800/12651 is reporting failure:1
>2018-07-11 10:47:55.817169 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.2 10.8.128.106:6800/12651
>2018-07-11 10:47:55.819576 7f29191ed700  1 mon.magna102@0(leader).osd e554 prepare_failure osd.10 10.8.128.107:6800/14975 from osd.6 10.8.128.115:6800/13323 is reporting failure:1
>2018-07-11 10:47:55.819590 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.10 10.8.128.107:6800/14975 reported failed by osd.6 10.8.128.115:6800/13323
>2018-07-11 10:47:56.794350 7f29149e4700  1 mon.magna102@0(leader).log v20853 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20853)
>2018-07-11 10:47:57.794426 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:47:57.944958 7f29149e4700  1 mon.magna102@0(leader).log v20854 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20854)
>2018-07-11 10:48:02.912726 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:48:03.403622 7f291b9f2700  1 mon.magna102@0(leader).osd e554  we have enough reporters to mark osd.10 down
>2018-07-11 10:48:03.403643 7f291b9f2700  0 log_channel(cluster) log [INF] : osd.10 failed (root=default,host=magna107) (6 reporters from different host after 27.584116 >= grace 26.201063)
>2018-07-11 10:48:03.404484 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 10:48:03.529076 7f29149e4700  1 mon.magna102@0(leader).osd e555 e555: 12 total, 11 up, 12 in
>2018-07-11 10:48:03.571318 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e555: 12 total, 11 up, 12 in
>2018-07-11 10:48:03.654301 7f29149e4700  1 mon.magna102@0(leader).log v20855 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20855)
>2018-07-11 10:48:03.746166 7f29149e4700  1 mon.magna102@0(leader).osd e556 e556: 12 total, 11 up, 12 in
>2018-07-11 10:48:03.788382 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e556: 12 total, 11 up, 12 in
>2018-07-11 10:48:04.738061 7f29149e4700  1 mon.magna102@0(leader).log v20856 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20856)
>2018-07-11 10:48:05.738109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:48:05.738156 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 112577/24217500 objects degraded (0.465%), 8 pgs degraded (PG_DEGRADED)
>2018-07-11 10:48:05.738177 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 10:48:05.738235 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 10:48:05.913479 7f29149e4700  1 mon.magna102@0(leader).log v20857 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 4 entries (version 20857)
>2018-07-11 10:48:11.054293 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.9 10.8.128.107:6804/12617 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054306 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.9 10.8.128.107:6804/12617 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.054355 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.0 10.8.128.113:6800/15255 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054366 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.113:6800/15255 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.054473 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.1 10.8.128.106:6804/12677 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054485 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.1 10.8.128.106:6804/12677 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.054786 7f29191ed700  1 mon.magna102@0(leader).osd e556  ignoring beacon from non-active osd.
>2018-07-11 10:48:11.054814 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.4 10.8.128.113:6808/15287 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054825 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.4 10.8.128.113:6808/15287 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.054871 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.11 10.8.128.114:6800/15919 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054882 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.11 10.8.128.114:6800/15919 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.054975 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.7 10.8.128.114:6804/15885 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.054986 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.7 10.8.128.114:6804/15885 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.055057 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.3 10.8.128.106:6801/12650 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.055068 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.106:6801/12650 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.055137 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.2 10.8.128.106:6800/12651 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.055148 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.2 10.8.128.106:6800/12651 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.055226 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.5 10.8.128.113:6804/15263 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.055238 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.5 10.8.128.113:6804/15263 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.055286 7f29191ed700  1 mon.magna102@0(leader).osd e556 prepare_failure osd.6 10.8.128.115:6800/13323 from osd.10 10.8.128.107:6800/14975 is reporting failure:0
>2018-07-11 10:48:11.055297 7f29191ed700  0 log_channel(cluster) log [DBG] : osd.6 10.8.128.115:6800/13323 failure report canceled by osd.10 10.8.128.107:6800/14975
>2018-07-11 10:48:11.180662 7f29149e4700  1 mon.magna102@0(leader).log v20858 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20858)
>2018-07-11 10:48:12.181550 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 10:48:12.264254 7f29149e4700  1 mon.magna102@0(leader).osd e557 e557: 12 total, 12 up, 12 in
>2018-07-11 10:48:12.306174 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/14975 boot
>2018-07-11 10:48:12.306244 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e557: 12 total, 12 up, 12 in
>2018-07-11 10:48:12.473016 7f29149e4700  1 mon.magna102@0(leader).log v20859 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20859)
>2018-07-11 10:48:12.556564 7f29149e4700  1 mon.magna102@0(leader).osd e558 e558: 12 total, 12 up, 12 in
>2018-07-11 10:48:12.598790 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e558: 12 total, 12 up, 12 in
>2018-07-11 10:48:13.404905 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 2 pgs inactive, 22 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:48:13.404938 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 474548/24217500 objects degraded (1.960%), 21 pgs degraded (PG_DEGRADED)
>2018-07-11 10:48:13.556812 7f29149e4700  1 mon.magna102@0(leader).log v20860 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20860)
>2018-07-11 10:48:17.157286 7f29149e4700  1 mon.magna102@0(leader).log v20861 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20861)
>2018-07-11 10:48:18.157286 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 199225/24217500 objects degraded (0.823%), 9 pgs degraded)
>2018-07-11 10:48:18.307649 7f29149e4700  1 mon.magna102@0(leader).log v20862 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20862)
>2018-07-11 10:48:18.405175 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:48:19.374587 7f29149e4700  1 mon.magna102@0(leader).log v20863 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20863)
>2018-07-11 10:48:19.391281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 10:48:20.441574 7f29149e4700  1 mon.magna102@0(leader).log v20864 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20864)
>2018-07-11 10:48:25.373234 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 52 slow requests are blocked > 32 sec. Implicated osds 0,1,8 (REQUEST_SLOW)
>2018-07-11 10:48:25.526866 7f29149e4700  1 mon.magna102@0(leader).log v20865 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20865)
>2018-07-11 10:48:33.410787 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 53 slow requests are blocked > 32 sec. Implicated osds 0,1,8 (REQUEST_SLOW)
>2018-07-11 10:48:33.527804 7f29149e4700  1 mon.magna102@0(leader).log v20866 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20866)
>2018-07-11 10:49:01.391258 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 0,1,8 (REQUEST_SLOW)
>2018-07-11 10:49:01.538135 7f29149e4700  1 mon.magna102@0(leader).log v20867 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20867)
>2018-07-11 10:49:02.912918 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:49:05.393327 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 6 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:05.546910 7f29149e4700  1 mon.magna102@0(leader).log v20868 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20868)
>2018-07-11 10:49:13.430635 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 17 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:13.547505 7f29149e4700  1 mon.magna102@0(leader).log v20869 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20869)
>2018-07-11 10:49:18.430852 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 27 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:18.531397 7f29149e4700  1 mon.magna102@0(leader).log v20870 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20870)
>2018-07-11 10:49:27.983524 7f29149e4700  1 mon.magna102@0(leader).osd e559 e559: 12 total, 12 up, 12 in
>2018-07-11 10:49:28.017210 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e559: 12 total, 12 up, 12 in
>2018-07-11 10:49:28.508837 7f29149e4700  1 mon.magna102@0(leader).osd e560 e560: 12 total, 12 up, 12 in
>2018-07-11 10:49:28.542587 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e560: 12 total, 12 up, 12 in
>2018-07-11 10:49:29.050727 7f29149e4700  1 mon.magna102@0(leader).log v20871 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20871)
>2018-07-11 10:49:29.576007 7f29149e4700  1 mon.magna102@0(leader).osd e561 e561: 12 total, 12 up, 12 in
>2018-07-11 10:49:29.609724 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e561: 12 total, 12 up, 12 in
>2018-07-11 10:49:30.050780 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 26 pgs inactive, 20 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:30.643114 7f29149e4700  1 mon.magna102@0(leader).log v20872 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20872)
>2018-07-11 10:49:31.643117 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 10:49:31.643164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 39 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:49:31.801825 7f29149e4700  1 mon.magna102@0(leader).log v20873 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20873)
>2018-07-11 10:49:32.552010 7f29149e4700  1 mon.magna102@0(leader).osd e562 e562: 12 total, 12 up, 12 in
>2018-07-11 10:49:32.594039 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e562: 12 total, 12 up, 12 in
>2018-07-11 10:49:33.635568 7f29149e4700  1 mon.magna102@0(leader).log v20874 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20874)
>2018-07-11 10:49:33.719191 7f29149e4700  1 mon.magna102@0(leader).osd e563 e563: 12 total, 12 up, 12 in
>2018-07-11 10:49:33.761231 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e563: 12 total, 12 up, 12 in
>2018-07-11 10:49:34.794501 7f29149e4700  1 mon.magna102@0(leader).log v20875 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20875)
>2018-07-11 10:49:35.794547 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 14 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:35.969890 7f29149e4700  1 mon.magna102@0(leader).log v20876 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20876)
>2018-07-11 10:49:37.419948 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 24 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 10:49:37.461697 7f29149e4700  1 mon.magna102@0(leader).log v20877 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20877)
>2018-07-11 10:49:38.545436 7f29149e4700  1 mon.magna102@0(leader).log v20878 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20878)
>2018-07-11 10:49:43.503806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:49:44.579457 7f29149e4700  1 mon.magna102@0(leader).log v20879 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20879)
>2018-07-11 10:49:51.416247 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:49:51.580200 7f29149e4700  1 mon.magna102@0(leader).log v20880 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20880)
>2018-07-11 10:49:57.419144 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 7 slow requests are blocked > 32 sec. Implicated osds 0,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:49:57.564103 7f29149e4700  1 mon.magna102@0(leader).log v20881 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20881)
>2018-07-11 10:50:02.913101 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 10:50:03.504617 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 11 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:04.557872 7f29149e4700  1 mon.magna102@0(leader).log v20882 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20882)
>2018-07-11 10:50:08.504871 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 13 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:08.625014 7f29149e4700  1 mon.magna102@0(leader).log v20883 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20883)
>2018-07-11 10:50:15.428091 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 19 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:15.584207 7f29149e4700  1 mon.magna102@0(leader).log v20884 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20884)
>2018-07-11 10:50:19.568108 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10526. Immutable memtables: 0.
>
>2018-07-11 10:50:19.568151 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 933] Syncing log #10523
>2018-07-11 10:50:19.584625 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.568141) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:50:19.584637 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 933] Flushing memtable with next log file: 10526
>
>2018-07-11 10:50:19.584651 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219584643, "job": 933, "event": "flush_started", "num_memtables": 1, "num_entries": 2084, "num_deletes": 251, "memory_usage": 4390952}
>2018-07-11 10:50:19.584654 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 933] Level-0 flush table #10527: started
>2018-07-11 10:50:19.668545 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219668517, "cf_name": "default", "job": 933, "event": "table_file_creation", "file_number": 10527, "file_size": 4075948, "table_properties": {"data_size": 4044861, "index_size": 16949, "filter_size": 13212, "raw_key_size": 16964, "raw_average_key_size": 20, "raw_value_size": 4021272, "raw_average_value_size": 4892, "num_data_blocks": 508, "num_entries": 822, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:50:19.668557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 933] Level-0 flush table #10527: 4075948 bytes OK
>2018-07-11 10:50:19.718596 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.685009) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10527 started
>2018-07-11 10:50:19.718604 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.718556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10527: memtable #1 done
>2018-07-11 10:50:19.718607 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.718571) EVENT_LOG_v1 {"time_micros": 1531306219718564, "job": 933, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:50:19.718608 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.718585) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:50:19.718628 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 933] Try to delete WAL files size 4342466, prev total WAL file size 4342466, number of live WAL files 2.
>
>2018-07-11 10:50:19.719679 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:50:19.719871 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 10:50:19.719878 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 68246.6 total, 901.4 interval
>Cumulative writes: 92K writes, 436K keys, 92K commit groups, 1.0 writes per commit group, ingest: 0.73 GB, 0.01 MB/s
>Cumulative WAL: 92K writes, 92K syncs, 1.00 writes per sync, written: 0.73 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1091 writes, 4891 keys, 1091 commit groups, 1.0 writes per commit group, ingest: 8.50 MB, 0.01 MB/s
>Interval WAL: 1091 writes, 1091 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.89 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        38       467    0.081       0      0
>  L1      1/1   14.31 MB   0.0      7.3     0.6      6.7       6.7     -0.0       0.0  10.7     72.6     66.4       104       465    0.223   1663K   199K
> Sum      2/2   18.20 MB   0.0      7.3     0.6      6.7       7.3      0.6       0.0  11.7     53.2     53.2       141       932    0.152   1663K   199K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.0     53.9     55.0         1         8    0.152     14K   1758
>Uptime(secs): 68246.6 total, 68246.6 interval
>Flush(GB): cumulative 0.629, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.35 GB write, 0.11 MB/s write, 7.35 GB read, 0.11 MB/s read, 141.4 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.89 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.0        38       467    0.081       0      0
>  L1      1/1   14.31 MB   0.0      7.3     0.6      6.7       6.7     -0.0       0.0  10.7     72.6     66.4       104       465    0.223   1663K   199K
> Sum      2/2   18.20 MB   0.0      7.3     0.6      6.7       7.3      0.6       0.0  11.7     53.2     53.2       141       932    0.152   1663K   199K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 68246.6 total, 68246.6 interval
>Flush(GB): cumulative 0.629, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.35 GB write, 0.11 MB/s write, 7.35 GB read, 0.11 MB/s read, 141.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 10:50:19.719905 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.719898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:50:19.719910 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 934] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:50:19.719916 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 933 Base level 0, inputs: [10527(3980KB)], [10525(14MB)]
>
>2018-07-11 10:50:19.719928 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219719919, "job": 934, "event": "compaction_started", "files_L0": [10527], "files_L1": [10525], "score": -1, "input_data_size": 19086019}
>2018-07-11 10:50:19.919285 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 934] Generated table #10528: 3429 keys, 16551633 bytes
>2018-07-11 10:50:19.919329 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219919314, "cf_name": "default", "job": 934, "event": "table_file_creation", "file_number": 10528, "file_size": 16551633, "table_properties": {"data_size": 16425735, "index_size": 71811, "filter_size": 53160, "raw_key_size": 69389, "raw_average_key_size": 20, "raw_value_size": 16326584, "raw_average_value_size": 4761, "num_data_blocks": 2092, "num_entries": 3429, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:50:19.960794 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 934] Compacted 1@0 + 1@1 files to L1 => 16551633 bytes
>2018-07-11 10:50:19.994335 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.994267) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 68.7 wr, level 1, files in(1, 1) out(1) MB in(3.9, 14.3) out(15.8), read-write-amplify(8.7) write-amplify(4.1) OK, records in: 3944, records dropped: 515
>
>2018-07-11 10:50:19.994344 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:50:19.994288) EVENT_LOG_v1 {"time_micros": 1531306219994279, "job": 934, "event": "compaction_finished", "compaction_time_micros": 240849, "output_level": 1, "num_output_files": 1, "total_output_size": 16551633, "num_input_records": 3944, "num_output_records": 3429, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:50:19.995147 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219995143, "job": 934, "event": "table_file_deletion", "file_number": 10527}
>2018-07-11 10:50:19.998141 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306219998139, "job": 934, "event": "table_file_deletion", "file_number": 10525}
>2018-07-11 10:50:21.431112 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:21.603365 7f29149e4700  1 mon.magna102@0(leader).log v20885 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20885)
>2018-07-11 10:50:31.436004 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 27 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:31.604305 7f29149e4700  1 mon.magna102@0(leader).log v20886 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20886)
>2018-07-11 10:50:35.438071 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs inactive, 10 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:50:35.588147 7f29149e4700  1 mon.magna102@0(leader).log v20887 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20887)
>2018-07-11 10:50:38.512267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 33 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:38.630212 7f29149e4700  1 mon.magna102@0(leader).log v20888 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20888)
>2018-07-11 10:50:43.512509 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 39 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:44.572426 7f29149e4700  1 mon.magna102@0(leader).log v20889 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20889)
>2018-07-11 10:50:48.512746 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 45 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:48.632472 7f29149e4700  1 mon.magna102@0(leader).log v20890 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20890)
>2018-07-11 10:50:53.512990 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 47 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:53.633217 7f29149e4700  1 mon.magna102@0(leader).log v20891 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20891)
>2018-07-11 10:50:58.513225 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 53 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:50:58.583642 7f29149e4700  1 mon.magna102@0(leader).log v20892 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20892)
>2018-07-11 10:51:02.913288 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:51:03.513456 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 55 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:51:03.634308 7f29149e4700  1 mon.magna102@0(leader).log v20893 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20893)
>2018-07-11 10:51:08.513684 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 55 slow requests are blocked > 32 sec. Implicated osds 0,1,10 (REQUEST_SLOW)
>2018-07-11 10:51:08.593065 7f29149e4700  1 mon.magna102@0(leader).log v20894 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20894)
>2018-07-11 10:51:09.593026 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 10 pgs inactive, 2 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:51:09.743391 7f29149e4700  1 mon.magna102@0(leader).log v20895 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20895)
>2018-07-11 10:51:13.513921 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:51:13.635548 7f29149e4700  1 mon.magna102@0(leader).log v20896 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20896)
>2018-07-11 10:51:25.462664 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,10 (REQUEST_SLOW)
>2018-07-11 10:51:25.619867 7f29149e4700  1 mon.magna102@0(leader).log v20897 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20897)
>2018-07-11 10:51:33.514681 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 9 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:51:33.637245 7f29149e4700  1 mon.magna102@0(leader).log v20898 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20898)
>2018-07-11 10:51:38.514907 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 13 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:51:38.604349 7f29149e4700  1 mon.magna102@0(leader).log v20899 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20899)
>2018-07-11 10:51:43.515152 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 23 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:51:43.639554 7f29149e4700  1 mon.magna102@0(leader).log v20900 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20900)
>2018-07-11 10:51:51.475323 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 27 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:51:51.648583 7f29149e4700  1 mon.magna102@0(leader).log v20901 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20901)
>2018-07-11 10:51:58.599338 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10529. Immutable memtables: 0.
>
>2018-07-11 10:51:58.599382 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 935] Syncing log #10526
>2018-07-11 10:51:58.615875 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.599372) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:51:58.615886 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 935] Flushing memtable with next log file: 10529
>
>2018-07-11 10:51:58.615899 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318615892, "job": 935, "event": "flush_started", "num_memtables": 1, "num_entries": 734, "num_deletes": 250, "memory_usage": 1204256}
>2018-07-11 10:51:58.615902 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 935] Level-0 flush table #10530: started
>2018-07-11 10:51:58.666395 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318666381, "cf_name": "default", "job": 935, "event": "table_file_creation", "file_number": 10530, "file_size": 809020, "table_properties": {"data_size": 802070, "index_size": 2973, "filter_size": 3058, "raw_key_size": 7878, "raw_average_key_size": 21, "raw_value_size": 794932, "raw_average_value_size": 2177, "num_data_blocks": 92, "num_entries": 365, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 10:51:58.666405 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 935] Level-0 flush table #10530: 809020 bytes OK
>2018-07-11 10:51:58.716423 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.682961) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10530 started
>2018-07-11 10:51:58.716432 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.716388) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10530: memtable #1 done
>2018-07-11 10:51:58.716434 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.716402) EVENT_LOG_v1 {"time_micros": 1531306318716396, "job": 935, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:51:58.716436 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.716414) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:51:58.716444 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 935] Try to delete WAL files size 1188930, prev total WAL file size 1188930, number of live WAL files 2.
>
>2018-07-11 10:51:58.716772 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:51:58.716825 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.716818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 10:51:58.716834 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 936] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:51:58.716840 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 935 Base level 0, inputs: [10530(790KB)], [10528(15MB)]
>
>2018-07-11 10:51:58.716852 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318716843, "job": 936, "event": "compaction_started", "files_L0": [10530], "files_L1": [10528], "score": -1, "input_data_size": 17360653}
>2018-07-11 10:51:58.908802 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 936] Generated table #10531: 3329 keys, 15934638 bytes
>2018-07-11 10:51:58.908845 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318908818, "cf_name": "default", "job": 936, "event": "table_file_creation", "file_number": 10531, "file_size": 15934638, "table_properties": {"data_size": 15815111, "index_size": 67641, "filter_size": 50959, "raw_key_size": 67004, "raw_average_key_size": 20, "raw_value_size": 15720381, "raw_average_value_size": 4722, "num_data_blocks": 1976, "num_entries": 3329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:51:58.941963 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 936] Compacted 1@0 + 1@1 files to L1 => 15934638 bytes
>2018-07-11 10:51:58.975499 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.975427) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.1 rd, 70.8 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.8) out(15.2), read-write-amplify(41.2) write-amplify(19.7) OK, records in: 3794, records dropped: 465
>
>2018-07-11 10:51:58.975508 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:51:58.975449) EVENT_LOG_v1 {"time_micros": 1531306318975439, "job": 936, "event": "compaction_finished", "compaction_time_micros": 225096, "output_level": 1, "num_output_files": 1, "total_output_size": 15934638, "num_input_records": 3794, "num_output_records": 3329, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:51:58.975719 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318975715, "job": 936, "event": "table_file_deletion", "file_number": 10530}
>2018-07-11 10:51:58.978635 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306318978633, "job": 936, "event": "table_file_deletion", "file_number": 10528}
>2018-07-11 10:51:58.978687 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:52:01.480452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 31 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:52:01.643259 7f29149e4700  1 mon.magna102@0(leader).log v20902 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20902)
>2018-07-11 10:52:02.913470 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:52:06.528854 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:52:06.528878 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3684998441' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:52:06.530110 7f29191ed700  1 mon.magna102@0(leader).log v20902 check_sub sending message to client.? 10.8.128.108:0/3684998441 with 1 entries (version 20902)
>2018-07-11 10:52:06.618650 7f29149e4700  1 mon.magna102@0(leader).log v20903 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20903)
>2018-07-11 10:52:06.618669 7f29149e4700  1 mon.magna102@0(leader).log v20903 check_sub sending message to client.? 10.8.128.108:0/3684998441 with 0 entries (version 20903)
>2018-07-11 10:52:07.618671 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 41 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:52:07.769049 7f29149e4700  1 mon.magna102@0(leader).log v20904 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20904)
>2018-07-11 10:52:07.769093 7f29149e4700  1 mon.magna102@0(leader).log v20904 check_sub sending message to client.? 10.8.128.108:0/3684998441 with 1 entries (version 20904)
>2018-07-11 10:52:13.557958 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,8 (REQUEST_SLOW)
>2018-07-11 10:52:14.619614 7f29149e4700  1 mon.magna102@0(leader).log v20905 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20905)
>2018-07-11 10:52:15.619631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 6 pgs inactive, 2 pgs peering (PG_AVAILABILITY)
>2018-07-11 10:52:15.786752 7f29149e4700  1 mon.magna102@0(leader).log v20906 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20906)
>2018-07-11 10:52:31.494678 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:52:31.664173 7f29149e4700  1 mon.magna102@0(leader).log v20907 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20907)
>2018-07-11 10:52:38.558885 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:52:38.664643 7f29149e4700  1 mon.magna102@0(leader).log v20908 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20908)
>2018-07-11 10:52:40.810538 7f29191ed700  1 mon.magna102@0(leader).auth v268 client did not provide supported auth type
>2018-07-11 10:52:43.559117 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,10 (REQUEST_SLOW)
>2018-07-11 10:52:43.673713 7f29149e4700  1 mon.magna102@0(leader).log v20909 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20909)
>2018-07-11 10:52:45.501363 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 2 pgs peering)
>2018-07-11 10:52:45.649035 7f29149e4700  1 mon.magna102@0(leader).log v20910 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20910)
>2018-07-11 10:52:48.559358 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 10:52:48.649341 7f29149e4700  1 mon.magna102@0(leader).log v20911 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20911)
>2018-07-11 10:52:59.507611 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,2,10 (REQUEST_SLOW)
>2018-07-11 10:52:59.658648 7f29149e4700  1 mon.magna102@0(leader).log v20912 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20912)
>2018-07-11 10:53:02.913648 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:53:07.511264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 0,2,8,10 (REQUEST_SLOW)
>2018-07-11 10:53:07.659371 7f29149e4700  1 mon.magna102@0(leader).log v20913 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20913)
>2018-07-11 10:53:47.529550 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 0,8,10 (REQUEST_SLOW)
>2018-07-11 10:53:47.680126 7f29149e4700  1 mon.magna102@0(leader).log v20914 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20914)
>2018-07-11 10:53:54.605687 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 nearfull osd(s) (OSD_NEARFULL)
>2018-07-11 10:53:54.605733 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 8 pool(s) nearfull (POOL_NEARFULL)
>2018-07-11 10:53:54.672384 7f29149e4700  1 mon.magna102@0(leader).osd e564 e564: 12 total, 12 up, 12 in
>2018-07-11 10:53:54.705961 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e564: 12 total, 12 up, 12 in
>2018-07-11 10:53:54.772673 7f29149e4700  1 mon.magna102@0(leader).log v20915 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 20915)
>2018-07-11 10:53:55.839455 7f29149e4700  1 mon.magna102@0(leader).log v20916 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20916)
>2018-07-11 10:54:02.913819 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:54:17.542779 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,10 (REQUEST_SLOW)
>2018-07-11 10:54:17.716430 7f29149e4700  1 mon.magna102@0(leader).log v20917 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20917)
>2018-07-11 10:54:18.799873 7f29149e4700  1 mon.magna102@0(leader).log v20918 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20918)
>2018-07-11 10:54:23.583527 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:54:23.708882 7f29149e4700  1 mon.magna102@0(leader).log v20919 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20919)
>2018-07-11 10:54:37.551483 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 5 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:54:37.702902 7f29149e4700  1 mon.magna102@0(leader).log v20920 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20920)
>2018-07-11 10:54:43.586456 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 7 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:54:43.703476 7f29149e4700  1 mon.magna102@0(leader).log v20921 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20921)
>2018-07-11 10:55:02.914009 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:55:02.959379 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:55:02.959400 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3425909011' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:55:03.079741 7f29149e4700  1 mon.magna102@0(leader).log v20922 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20922)
>2018-07-11 10:55:05.228900 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 10:55:05.228919 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/2225600585' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 10:55:05.346748 7f29149e4700  1 mon.magna102@0(leader).log v20923 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20923)
>2018-07-11 10:55:07.564664 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 7 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:55:07.713953 7f29149e4700  1 mon.magna102@0(leader).log v20924 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20924)
>2018-07-11 10:55:21.570786 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:55:21.723371 7f29149e4700  1 mon.magna102@0(leader).log v20925 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20925)
>2018-07-11 10:55:28.606877 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 10:55:28.707268 7f29149e4700  1 mon.magna102@0(leader).log v20926 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20926)
>2018-07-11 10:56:02.914199 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:56:37.604097 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:56:37.755189 7f29149e4700  1 mon.magna102@0(leader).log v20927 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20927)
>2018-07-11 10:57:02.914385 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 10:57:07.617319 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 10:57:07.766864 7f29149e4700  1 mon.magna102@0(leader).log v20928 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20928)
>2018-07-11 10:57:07.833946 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10532. Immutable memtables: 0.
>
>2018-07-11 10:57:07.833991 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 937] Syncing log #10529
>2018-07-11 10:57:07.850497 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.833981) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 10:57:07.850509 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 937] Flushing memtable with next log file: 10532
>
>2018-07-11 10:57:07.850536 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306627850527, "job": 937, "event": "flush_started", "num_memtables": 1, "num_entries": 1567, "num_deletes": 251, "memory_usage": 3001816}
>2018-07-11 10:57:07.850539 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 937] Level-0 flush table #10533: started
>2018-07-11 10:57:07.917715 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306627917700, "cf_name": "default", "job": 937, "event": "table_file_creation", "file_number": 10533, "file_size": 2842730, "table_properties": {"data_size": 2820051, "index_size": 12231, "filter_size": 9524, "raw_key_size": 13573, "raw_average_key_size": 20, "raw_value_size": 2802104, "raw_average_value_size": 4258, "num_data_blocks": 369, "num_entries": 658, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 10:57:07.917726 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 937] Level-0 flush table #10533: 2842730 bytes OK
>2018-07-11 10:57:07.967629 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.934205) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10533 started
>2018-07-11 10:57:07.967637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.967591) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10533: memtable #1 done
>2018-07-11 10:57:07.967639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.967606) EVENT_LOG_v1 {"time_micros": 1531306627967599, "job": 937, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 10:57:07.967641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.967618) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 10:57:07.967660 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 937] Try to delete WAL files size 2970696, prev total WAL file size 2970696, number of live WAL files 2.
>
>2018-07-11 10:57:07.968399 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 10:57:07.968449 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:57:07.968442) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 10:57:07.968459 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 938] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 10:57:07.968465 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 937 Base level 0, inputs: [10533(2776KB)], [10531(15MB)]
>
>2018-07-11 10:57:07.968479 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306627968469, "job": 938, "event": "compaction_started", "files_L0": [10533], "files_L1": [10531], "score": -1, "input_data_size": 18777368}
>2018-07-11 10:57:08.160069 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 938] Generated table #10534: 3473 keys, 16782099 bytes
>2018-07-11 10:57:08.160119 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306628160091, "cf_name": "default", "job": 938, "event": "table_file_creation", "file_number": 10534, "file_size": 16782099, "table_properties": {"data_size": 16655575, "index_size": 71883, "filter_size": 53714, "raw_key_size": 70222, "raw_average_key_size": 20, "raw_value_size": 16555759, "raw_average_value_size": 4766, "num_data_blocks": 2093, "num_entries": 3473, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 10:57:08.193232 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 938] Compacted 1@0 + 1@1 files to L1 => 16782099 bytes
>2018-07-11 10:57:08.226790 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:57:08.226724) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.6 rd, 74.7 wr, level 1, files in(1, 1) out(1) MB in(2.7, 15.2) out(16.0), read-write-amplify(12.5) write-amplify(5.9) OK, records in: 3987, records dropped: 514
>
>2018-07-11 10:57:08.226799 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-10:57:08.226745) EVENT_LOG_v1 {"time_micros": 1531306628226735, "job": 938, "event": "compaction_finished", "compaction_time_micros": 224736, "output_level": 1, "num_output_files": 1, "total_output_size": 16782099, "num_input_records": 3987, "num_output_records": 3473, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 10:57:08.227447 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306628227442, "job": 938, "event": "table_file_deletion", "file_number": 10533}
>2018-07-11 10:57:08.230559 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306628230557, "job": 938, "event": "table_file_deletion", "file_number": 10531}
>2018-07-11 10:57:21.623527 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:57:21.795073 7f29149e4700  1 mon.magna102@0(leader).log v20929 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20929)
>2018-07-11 10:57:28.661790 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 10:57:28.787304 7f29149e4700  1 mon.magna102@0(leader).log v20930 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20930)
>2018-07-11 10:58:02.914542 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 10:58:41.658669 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 10:58:41.811339 7f29149e4700  1 mon.magna102@0(leader).log v20931 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20931)
>2018-07-11 10:58:43.659522 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 10:58:43.829554 7f29149e4700  1 mon.magna102@0(leader).log v20932 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20932)
>2018-07-11 10:59:01.667362 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 10:59:01.822560 7f29149e4700  1 mon.magna102@0(leader).log v20933 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20933)
>2018-07-11 10:59:02.914724 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 10:59:08.705562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 10:59:08.806457 7f29149e4700  1 mon.magna102@0(leader).log v20934 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20934)
>2018-07-11 11:00:02.914920 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:00:11.698110 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:00:11.853999 7f29149e4700  1 mon.magna102@0(leader).log v20935 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20935)
>2018-07-11 11:00:13.698930 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 11:00:13.862655 7f29149e4700  1 mon.magna102@0(leader).log v20936 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20936)
>2018-07-11 11:00:18.813304 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10535. Immutable memtables: 0.
>
>2018-07-11 11:00:18.813349 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 939] Syncing log #10532
>2018-07-11 11:00:18.829789 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.813339) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:00:18.829801 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 939] Flushing memtable with next log file: 10535
>
>2018-07-11 11:00:18.829827 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306818829819, "job": 939, "event": "flush_started", "num_memtables": 1, "num_entries": 984, "num_deletes": 250, "memory_usage": 1635712}
>2018-07-11 11:00:18.829830 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 939] Level-0 flush table #10536: started
>2018-07-11 11:00:18.880322 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306818880308, "cf_name": "default", "job": 939, "event": "table_file_creation", "file_number": 10536, "file_size": 943091, "table_properties": {"data_size": 934816, "index_size": 3810, "filter_size": 3546, "raw_key_size": 8249, "raw_average_key_size": 21, "raw_value_size": 926813, "raw_average_value_size": 2413, "num_data_blocks": 119, "num_entries": 384, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:00:18.880332 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 939] Level-0 flush table #10536: 943091 bytes OK
>2018-07-11 11:00:18.938680 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.896862) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10536 started
>2018-07-11 11:00:18.938688 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.938645) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10536: memtable #1 done
>2018-07-11 11:00:18.938691 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.938659) EVENT_LOG_v1 {"time_micros": 1531306818938653, "job": 939, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:00:18.938706 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.938670) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:00:18.938712 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 939] Try to delete WAL files size 1615924, prev total WAL file size 1615924, number of live WAL files 2.
>
>2018-07-11 11:00:18.939258 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:00:18.939311 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:00:18.939303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:00:18.939320 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 940] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:00:18.939327 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 939 Base level 0, inputs: [10536(920KB)], [10534(16MB)]
>
>2018-07-11 11:00:18.939339 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306818939330, "job": 940, "event": "compaction_started", "files_L0": [10536], "files_L1": [10534], "score": -1, "input_data_size": 17725190}
>2018-07-11 11:00:19.131056 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 940] Generated table #10537: 3438 keys, 16615288 bytes
>2018-07-11 11:00:19.131103 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306819131076, "cf_name": "default", "job": 940, "event": "table_file_creation", "file_number": 10537, "file_size": 16615288, "table_properties": {"data_size": 16491115, "index_size": 70212, "filter_size": 53034, "raw_key_size": 69220, "raw_average_key_size": 20, "raw_value_size": 16393035, "raw_average_value_size": 4768, "num_data_blocks": 2051, "num_entries": 3438, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:00:19.164243 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 940] Compacted 1@0 + 1@1 files to L1 => 16615288 bytes
>2018-07-11 11:00:19.206090 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:00:19.206018) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.8 rd, 73.9 wr, level 1, files in(1, 1) out(1) MB in(0.9, 16.0) out(15.8), read-write-amplify(36.4) write-amplify(17.6) OK, records in: 3857, records dropped: 419
>
>2018-07-11 11:00:19.206100 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:00:19.206040) EVENT_LOG_v1 {"time_micros": 1531306819206030, "job": 940, "event": "compaction_finished", "compaction_time_micros": 224890, "output_level": 1, "num_output_files": 1, "total_output_size": 16615288, "num_input_records": 3857, "num_output_records": 3438, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:00:19.206369 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306819206365, "job": 940, "event": "table_file_deletion", "file_number": 10536}
>2018-07-11 11:00:19.209341 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531306819209338, "job": 940, "event": "table_file_deletion", "file_number": 10534}
>2018-07-11 11:00:19.209385 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:00:33.707651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:00:33.857794 7f29149e4700  1 mon.magna102@0(leader).log v20937 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20937)
>2018-07-11 11:00:38.780435 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 11:00:38.900319 7f29149e4700  1 mon.magna102@0(leader).log v20938 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20938)
>2018-07-11 11:01:02.915120 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:01:47.740255 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 11:01:47.889712 7f29149e4700  1 mon.magna102@0(leader).log v20939 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20939)
>2018-07-11 11:01:51.741960 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 11:01:51.890131 7f29149e4700  1 mon.magna102@0(leader).log v20940 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20940)
>2018-07-11 11:02:02.915318 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:02:07.748844 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 11:02:07.900932 7f29149e4700  1 mon.magna102@0(leader).log v20941 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20941)
>2018-07-11 11:03:02.915499 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:03:22.918367 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:03:22.918393 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3380075755' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:03:22.919642 7f29191ed700  1 mon.magna102@0(leader).log v20941 check_sub sending message to client.? 10.8.128.102:0/3380075755 with 1 entries (version 20941)
>2018-07-11 11:03:23.032269 7f29149e4700  1 mon.magna102@0(leader).log v20942 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20942)
>2018-07-11 11:03:23.032289 7f29149e4700  1 mon.magna102@0(leader).log v20942 check_sub sending message to client.? 10.8.128.102:0/3380075755 with 0 entries (version 20942)
>2018-07-11 11:03:27.784236 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 2,10)
>2018-07-11 11:03:27.941240 7f29149e4700  1 mon.magna102@0(leader).log v20943 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20943)
>2018-07-11 11:03:41.790322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 11:03:41.942324 7f29149e4700  1 mon.magna102@0(leader).log v20944 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20944)
>2018-07-11 11:03:48.825793 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 11:03:48.942849 7f29149e4700  1 mon.magna102@0(leader).log v20945 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20945)
>2018-07-11 11:03:57.152021 7f29149e4700  1 mon.magna102@0(leader).log v20946 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20946)
>2018-07-11 11:04:02.915687 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:04:07.152841 7f29149e4700  1 mon.magna102@0(leader).log v20947 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20947)
>2018-07-11 11:04:47.982219 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10538. Immutable memtables: 0.
>
>2018-07-11 11:04:47.982265 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 941] Syncing log #10535
>2018-07-11 11:04:47.998765 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:04:47.982255) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:04:47.998777 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 941] Flushing memtable with next log file: 10538
>
>2018-07-11 11:04:47.998791 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307087998783, "job": 941, "event": "flush_started", "num_memtables": 1, "num_entries": 1289, "num_deletes": 251, "memory_usage": 2307408}
>2018-07-11 11:04:47.998794 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 941] Level-0 flush table #10539: started
>2018-07-11 11:04:48.057631 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307088057616, "cf_name": "default", "job": 941, "event": "table_file_creation", "file_number": 10539, "file_size": 2191529, "table_properties": {"data_size": 2173384, "index_size": 9670, "filter_size": 7551, "raw_key_size": 11588, "raw_average_key_size": 20, "raw_value_size": 2158615, "raw_average_value_size": 3834, "num_data_blocks": 294, "num_entries": 563, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:04:48.057642 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 941] Level-0 flush table #10539: 2191529 bytes OK
>2018-07-11 11:04:48.107687 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.074187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10539 started
>2018-07-11 11:04:48.107696 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.107650) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10539: memtable #1 done
>2018-07-11 11:04:48.107698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.107665) EVENT_LOG_v1 {"time_micros": 1531307088107658, "job": 941, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:04:48.107700 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.107677) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:04:48.107707 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 941] Try to delete WAL files size 2281422, prev total WAL file size 2281422, number of live WAL files 2.
>
>2018-07-11 11:04:48.108285 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:04:48.108464 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 11:04:48.108471 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 69115.0 total, 868.4 interval
>Cumulative writes: 93K writes, 441K keys, 93K commit groups, 1.0 writes per commit group, ingest: 0.73 GB, 0.01 MB/s
>Cumulative WAL: 93K writes, 93K syncs, 1.00 writes per sync, written: 0.73 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1005 writes, 4574 keys, 1005 commit groups, 1.0 writes per commit group, ingest: 7.68 MB, 0.01 MB/s
>Interval WAL: 1005 writes, 1005 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.09 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.1        38       471    0.081       0      0
>  L1      1/1   15.85 MB   0.0      7.4     0.6      6.8       6.8      0.0       0.0  10.7     72.6     66.4       105       469    0.223   1678K   201K
> Sum      2/2   17.94 MB   0.0      7.4     0.6      6.8       7.4      0.6       0.0  11.7     53.2     53.3       143       940    0.152   1678K   201K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.7     57.5     57.3         1         8    0.151     15K   1913
>Uptime(secs): 69115.0 total, 69115.0 interval
>Flush(GB): cumulative 0.635, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.42 GB write, 0.11 MB/s write, 7.42 GB read, 0.11 MB/s read, 142.6 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.09 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.1        38       471    0.081       0      0
>  L1      1/1   15.85 MB   0.0      7.4     0.6      6.8       6.8      0.0       0.0  10.7     72.6     66.4       105       469    0.223   1678K   201K
> Sum      2/2   17.94 MB   0.0      7.4     0.6      6.8       7.4      0.6       0.0  11.7     53.2     53.3       143       940    0.152   1678K   201K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 69115.0 total, 69115.0 interval
>Flush(GB): cumulative 0.635, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.42 GB write, 0.11 MB/s write, 7.42 GB read, 0.11 MB/s read, 142.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 11:04:48.108491 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.108484) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:04:48.108495 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 942] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:04:48.108501 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 941 Base level 0, inputs: [10539(2140KB)], [10537(15MB)]
>
>2018-07-11 11:04:48.108513 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307088108504, "job": 942, "event": "compaction_started", "files_L0": [10539], "files_L1": [10537], "score": -1, "input_data_size": 18806817}
>2018-07-11 11:04:48.308380 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 942] Generated table #10540: 3491 keys, 16608557 bytes
>2018-07-11 11:04:48.308416 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307088308401, "cf_name": "default", "job": 942, "event": "table_file_creation", "file_number": 10540, "file_size": 16608557, "table_properties": {"data_size": 16482321, "index_size": 71895, "filter_size": 53414, "raw_key_size": 70561, "raw_average_key_size": 20, "raw_value_size": 16382245, "raw_average_value_size": 4692, "num_data_blocks": 2093, "num_entries": 3491, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:04:48.349906 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 942] Compacted 1@0 + 1@1 files to L1 => 16608557 bytes
>2018-07-11 11:04:48.383402 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.383331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.9 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(2.1, 15.8) out(15.8), read-write-amplify(16.2) write-amplify(7.6) OK, records in: 4001, records dropped: 510
>
>2018-07-11 11:04:48.383410 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:04:48.383353) EVENT_LOG_v1 {"time_micros": 1531307088383343, "job": 942, "event": "compaction_finished", "compaction_time_micros": 241376, "output_level": 1, "num_output_files": 1, "total_output_size": 16608557, "num_input_records": 4001, "num_output_records": 3491, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:04:48.383902 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307088383898, "job": 942, "event": "table_file_deletion", "file_number": 10539}
>2018-07-11 11:04:48.387098 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307088387095, "job": 942, "event": "table_file_deletion", "file_number": 10537}
>2018-07-11 11:04:57.824916 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:04:57.976324 7f29149e4700  1 mon.magna102@0(leader).log v20948 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20948)
>2018-07-11 11:05:02.915839 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:05:03.827508 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 2)
>2018-07-11 11:05:03.976886 7f29149e4700  1 mon.magna102@0(leader).log v20949 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20949)
>2018-07-11 11:05:07.145047 7f29149e4700  1 mon.magna102@0(leader).log v20950 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20950)
>2018-07-11 11:05:09.129314 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:05:09.129340 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3777648225' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:05:09.130553 7f29191ed700  1 mon.magna102@0(leader).log v20950 check_sub sending message to client.? 10.8.128.102:0/3777648225 with 1 entries (version 20950)
>2018-07-11 11:05:09.278706 7f29149e4700  1 mon.magna102@0(leader).log v20951 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20951)
>2018-07-11 11:05:09.278726 7f29149e4700  1 mon.magna102@0(leader).log v20951 check_sub sending message to client.? 10.8.128.102:0/3777648225 with 0 entries (version 20951)
>2018-07-11 11:05:17.833763 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:05:17.987931 7f29149e4700  1 mon.magna102@0(leader).log v20952 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20952)
>2018-07-11 11:05:17.987975 7f29149e4700  1 mon.magna102@0(leader).log v20952 check_sub sending message to client.? 10.8.128.102:0/3777648225 with 1 entries (version 20952)
>2018-07-11 11:05:23.871488 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 11:05:23.988701 7f29149e4700  1 mon.magna102@0(leader).log v20953 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20953)
>2018-07-11 11:05:31.114122 7f29149e4700  1 mon.magna102@0(leader).log v20954 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20954)
>2018-07-11 11:05:41.140039 7f29149e4700  1 mon.magna102@0(leader).log v20955 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20955)
>2018-07-11 11:05:43.648755 7f29149e4700  1 mon.magna102@0(leader).log v20956 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20956)
>2018-07-11 11:05:51.141115 7f29149e4700  1 mon.magna102@0(leader).log v20957 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20957)
>2018-07-11 11:06:02.916023 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:06:26.000889 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:06:26.000911 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/2142809174' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:06:27.028246 7f29149e4700  1 mon.magna102@0(leader).log v20958 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20958)
>2018-07-11 11:06:27.737164 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:06:27.737206 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1832861973' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:06:28.103410 7f29149e4700  1 mon.magna102@0(leader).log v20959 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20959)
>2018-07-11 11:06:29.095521 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:06:29.095540 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/1058936161' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:06:29.245443 7f29149e4700  1 mon.magna102@0(leader).log v20960 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20960)
>2018-07-11 11:06:37.146227 7f29149e4700  1 mon.magna102@0(leader).log v20961 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20961)
>2018-07-11 11:06:38.146221 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 2,10)
>2018-07-11 11:06:38.296554 7f29149e4700  1 mon.magna102@0(leader).log v20962 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20962)
>2018-07-11 11:06:50.398125 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 11:06:50.398149 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/4080407540' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 11:06:51.005803 7f29149e4700  1 mon.magna102@0(leader).log v20963 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20963)
>2018-07-11 11:06:53.877128 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:06:54.023119 7f29149e4700  1 mon.magna102@0(leader).log v20964 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20964)
>2018-07-11 11:06:58.906293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 11:06:59.015215 7f29149e4700  1 mon.magna102@0(leader).log v20965 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20965)
>2018-07-11 11:07:01.148731 7f29149e4700  1 mon.magna102@0(leader).log v20966 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20966)
>2018-07-11 11:07:02.916171 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:07:05.320824 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:07:05.320842 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/2735689605' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:07:05.440919 7f29149e4700  1 mon.magna102@0(leader).log v20967 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20967)
>2018-07-11 11:07:34.979642 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:07:34.979661 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.108:0/3241050242' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:07:35.102709 7f29149e4700  1 mon.magna102@0(leader).log v20968 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20968)
>2018-07-11 11:07:43.899256 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 11:07:44.053687 7f29149e4700  1 mon.magna102@0(leader).log v20969 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20969)
>2018-07-11 11:07:47.153877 7f29149e4700  1 mon.magna102@0(leader).log v20970 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20970)
>2018-07-11 11:07:48.187306 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 11:07:48.220615 7f29149e4700  1 mon.magna102@0(leader).log v20971 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20971)
>2018-07-11 11:07:49.287818 7f29149e4700  1 mon.magna102@0(leader).log v20972 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20972)
>2018-07-11 11:08:02.916369 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:08:39.018195 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10541. Immutable memtables: 0.
>
>2018-07-11 11:08:39.018253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 943] Syncing log #10538
>2018-07-11 11:08:39.034709 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.018243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:08:39.034720 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 943] Flushing memtable with next log file: 10541
>
>2018-07-11 11:08:39.034747 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319034726, "job": 943, "event": "flush_started", "num_memtables": 1, "num_entries": 1260, "num_deletes": 250, "memory_usage": 2251664}
>2018-07-11 11:08:39.034750 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 943] Level-0 flush table #10542: started
>2018-07-11 11:08:39.085249 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319085233, "cf_name": "default", "job": 943, "event": "table_file_creation", "file_number": 10542, "file_size": 1406882, "table_properties": {"data_size": 1395373, "index_size": 5613, "filter_size": 4976, "raw_key_size": 9686, "raw_average_key_size": 21, "raw_value_size": 1385068, "raw_average_value_size": 3044, "num_data_blocks": 174, "num_entries": 455, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:08:39.085259 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 943] Level-0 flush table #10542: 1406882 bytes OK
>2018-07-11 11:08:39.135242 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.101788) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10542 started
>2018-07-11 11:08:39.135250 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.135206) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10542: memtable #1 done
>2018-07-11 11:08:39.135253 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.135220) EVENT_LOG_v1 {"time_micros": 1531307319135214, "job": 943, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:08:39.135255 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.135232) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:08:39.135261 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 943] Try to delete WAL files size 2226401, prev total WAL file size 2226401, number of live WAL files 2.
>
>2018-07-11 11:08:39.135847 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:08:39.135902 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.135895) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:08:39.135911 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 944] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:08:39.135917 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 943 Base level 0, inputs: [10542(1373KB)], [10540(15MB)]
>
>2018-07-11 11:08:39.135930 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319135920, "job": 944, "event": "compaction_started", "files_L0": [10542], "files_L1": [10540], "score": -1, "input_data_size": 18015439}
>2018-07-11 11:08:39.335943 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 944] Generated table #10543: 3547 keys, 17043598 bytes
>2018-07-11 11:08:39.335993 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319335965, "cf_name": "default", "job": 944, "event": "table_file_creation", "file_number": 10543, "file_size": 17043598, "table_properties": {"data_size": 16915317, "index_size": 72749, "filter_size": 54605, "raw_key_size": 71436, "raw_average_key_size": 20, "raw_value_size": 16813923, "raw_average_value_size": 4740, "num_data_blocks": 2125, "num_entries": 3547, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:08:39.377375 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 944] Compacted 1@0 + 1@1 files to L1 => 17043598 bytes
>2018-07-11 11:08:39.410987 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.410918) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.6 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.8) out(16.3), read-write-amplify(24.9) write-amplify(12.1) OK, records in: 3946, records dropped: 399
>
>2018-07-11 11:08:39.410996 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:08:39.410940) EVENT_LOG_v1 {"time_micros": 1531307319410930, "job": 944, "event": "compaction_finished", "compaction_time_micros": 241426, "output_level": 1, "num_output_files": 1, "total_output_size": 17043598, "num_input_records": 3946, "num_output_records": 3547, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:08:39.411329 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319411325, "job": 944, "event": "table_file_deletion", "file_number": 10542}
>2018-07-11 11:08:39.414278 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307319414275, "job": 944, "event": "table_file_deletion", "file_number": 10540}
>2018-07-11 11:08:39.414309 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:09:02.916552 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:10:02.809131 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:02.809154 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3431516910' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:02.916733 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:10:03.093499 7f29149e4700  1 mon.magna102@0(leader).log v20973 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20973)
>2018-07-11 11:10:04.970786 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:04.970810 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4019411132' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:05.227207 7f29149e4700  1 mon.magna102@0(leader).log v20974 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20974)
>2018-07-11 11:10:06.384854 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:06.384876 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3985666208' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:07.360859 7f29149e4700  1 mon.magna102@0(leader).log v20975 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20975)
>2018-07-11 11:10:07.791365 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:07.791387 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1059497358' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:08.494509 7f29149e4700  1 mon.magna102@0(leader).log v20976 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20976)
>2018-07-11 11:10:09.201007 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:09.201031 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3974085135' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:09.561385 7f29149e4700  1 mon.magna102@0(leader).log v20977 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20977)
>2018-07-11 11:10:12.724194 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:12.724218 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3899470673' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:13.129769 7f29149e4700  1 mon.magna102@0(leader).log v20978 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20978)
>2018-07-11 11:10:15.763467 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:10:15.763493 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3818791026' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:10:15.764867 7f29191ed700  1 mon.magna102@0(leader).log v20978 check_sub sending message to client.? 10.8.128.102:0/3818791026 with 1 entries (version 20978)
>2018-07-11 11:10:15.896754 7f29149e4700  1 mon.magna102@0(leader).log v20979 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20979)
>2018-07-11 11:10:15.896774 7f29149e4700  1 mon.magna102@0(leader).log v20979 check_sub sending message to client.? 10.8.128.102:0/3818791026 with 0 entries (version 20979)
>2018-07-11 11:11:02.917018 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:11:21.127840 7f29149e4700  1 mon.magna102@0(leader).log v20980 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20980)
>2018-07-11 11:12:00.172615 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10544. Immutable memtables: 0.
>
>2018-07-11 11:12:00.172660 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 945] Syncing log #10541
>2018-07-11 11:12:00.189190 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.172650) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:12:00.189202 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 945] Flushing memtable with next log file: 10544
>
>2018-07-11 11:12:00.189229 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520189221, "job": 945, "event": "flush_started", "num_memtables": 1, "num_entries": 1027, "num_deletes": 251, "memory_usage": 1685064}
>2018-07-11 11:12:00.189232 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 945] Level-0 flush table #10545: started
>2018-07-11 11:12:00.239732 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520239718, "cf_name": "default", "job": 945, "event": "table_file_creation", "file_number": 10545, "file_size": 1611891, "table_properties": {"data_size": 1597994, "index_size": 7224, "filter_size": 5753, "raw_key_size": 9977, "raw_average_key_size": 20, "raw_value_size": 1586075, "raw_average_value_size": 3263, "num_data_blocks": 220, "num_entries": 486, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:12:00.239742 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 945] Level-0 flush table #10545: 1611891 bytes OK
>2018-07-11 11:12:00.289736 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.256288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10545 started
>2018-07-11 11:12:00.289745 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.289699) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10545: memtable #1 done
>2018-07-11 11:12:00.289748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.289714) EVENT_LOG_v1 {"time_micros": 1531307520289708, "job": 945, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:12:00.289750 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.289726) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:12:00.289756 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 945] Try to delete WAL files size 1663994, prev total WAL file size 1663994, number of live WAL files 2.
>
>2018-07-11 11:12:00.290193 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:12:00.290246 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.290239) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:12:00.290256 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 946] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:12:00.290262 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 945 Base level 0, inputs: [10545(1574KB)], [10543(16MB)]
>
>2018-07-11 11:12:00.290275 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520290265, "job": 946, "event": "compaction_started", "files_L0": [10545], "files_L1": [10543], "score": -1, "input_data_size": 18655489}
>2018-07-11 11:12:00.490451 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 946] Generated table #10546: 3523 keys, 16546355 bytes
>2018-07-11 11:12:00.490499 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520490484, "cf_name": "default", "job": 946, "event": "table_file_creation", "file_number": 10546, "file_size": 16546355, "table_properties": {"data_size": 16420103, "index_size": 71955, "filter_size": 53370, "raw_key_size": 71166, "raw_average_key_size": 20, "raw_value_size": 16319537, "raw_average_value_size": 4632, "num_data_blocks": 2094, "num_entries": 3523, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:12:00.531944 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 946] Compacted 1@0 + 1@1 files to L1 => 16546355 bytes
>2018-07-11 11:12:00.573785 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.573718) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.2 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(1.5, 16.3) out(15.8), read-write-amplify(21.8) write-amplify(10.3) OK, records in: 4033, records dropped: 510
>
>2018-07-11 11:12:00.573794 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:12:00.573740) EVENT_LOG_v1 {"time_micros": 1531307520573730, "job": 946, "event": "compaction_finished", "compaction_time_micros": 241653, "output_level": 1, "num_output_files": 1, "total_output_size": 16546355, "num_input_records": 4033, "num_output_records": 3523, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:12:00.574149 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520574145, "job": 946, "event": "table_file_deletion", "file_number": 10545}
>2018-07-11 11:12:00.577544 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307520577541, "job": 946, "event": "table_file_deletion", "file_number": 10543}
>2018-07-11 11:12:02.917191 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:13:02.917348 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:13:35.413527 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:13:35.413550 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1019933121' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:13:35.532986 7f29149e4700  1 mon.magna102@0(leader).log v20981 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20981)
>2018-07-11 11:13:37.100929 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:13:37.100953 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3121133204' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:13:37.666646 7f29149e4700  1 mon.magna102@0(leader).log v20982 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20982)
>2018-07-11 11:13:38.651776 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:13:38.651802 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2603594028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:13:38.653129 7f29191ed700  1 mon.magna102@0(leader).log v20982 check_sub sending message to client.? 10.8.128.102:0/2603594028 with 1 entries (version 20982)
>2018-07-11 11:13:38.808650 7f29149e4700  1 mon.magna102@0(leader).log v20983 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20983)
>2018-07-11 11:13:38.808670 7f29149e4700  1 mon.magna102@0(leader).log v20983 check_sub sending message to client.? 10.8.128.102:0/2603594028 with 0 entries (version 20983)
>2018-07-11 11:13:53.659782 7f29149e4700  1 mon.magna102@0(leader).log v20984 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20984)
>2018-07-11 11:13:53.659815 7f29149e4700  1 mon.magna102@0(leader).log v20984 check_sub sending message to client.? 10.8.128.102:0/2603594028 with 0 entries (version 20984)
>2018-07-11 11:14:02.917526 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:14:14.071045 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:14:14.219826 7f29149e4700  1 mon.magna102@0(leader).log v20985 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20985)
>2018-07-11 11:14:14.219871 7f29149e4700  1 mon.magna102@0(leader).log v20985 check_sub sending message to client.? 10.8.128.102:0/2603594028 with 1 entries (version 20985)
>2018-07-11 11:14:19.103030 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:14:19.203578 7f29149e4700  1 mon.magna102@0(leader).log v20986 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20986)
>2018-07-11 11:14:19.203622 7f29149e4700  1 mon.magna102@0(leader).log v20986 check_sub sending message to client.? 10.8.128.102:0/2603594028 with 1 entries (version 20986)
>2018-07-11 11:14:23.465554 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:14:23.465579 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3081903045' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:14:23.579039 7f29149e4700  1 mon.magna102@0(leader).log v20987 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20987)
>2018-07-11 11:14:25.970973 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:14:25.970997 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3937345667' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:14:26.087713 7f29149e4700  1 mon.magna102@0(leader).log v20988 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20988)
>2018-07-11 11:14:31.125815 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:14:31.125840 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/720705492' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:14:31.213357 7f29149e4700  1 mon.magna102@0(leader).log v20989 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20989)
>2018-07-11 11:15:02.917715 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:15:22.101807 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 11:15:22.251623 7f29149e4700  1 mon.magna102@0(leader).log v20990 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20990)
>2018-07-11 11:15:26.103516 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 11:15:26.260463 7f29149e4700  1 mon.magna102@0(leader).log v20991 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20991)
>2018-07-11 11:15:46.112154 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 11:15:46.288166 7f29149e4700  1 mon.magna102@0(leader).log v20992 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20992)
>2018-07-11 11:15:50.823225 7f29191ed700  0 log_channel(cluster) log [INF] : osd.10 marked itself down
>2018-07-11 11:15:50.900213 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:15:50.900236 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2458409478' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:15:51.239476 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN)
>2018-07-11 11:15:51.239524 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_NEARFULL (was: 1 nearfull osd(s))
>2018-07-11 11:15:51.239555 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: POOL_NEARFULL (was: 8 pool(s) nearfull)
>2018-07-11 11:15:51.280234 7f29149e4700  1 mon.magna102@0(leader).log v20993 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20993)
>2018-07-11 11:15:51.363971 7f29149e4700  1 mon.magna102@0(leader).osd e565 e565: 12 total, 11 up, 12 in
>2018-07-11 11:15:51.405856 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e565: 12 total, 11 up, 12 in
>2018-07-11 11:15:52.364114 7f29149e4700  1 mon.magna102@0(leader).log v20994 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 20994)
>2018-07-11 11:15:53.486956 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:15:53.486985 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/922841324' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:15:53.606160 7f29149e4700  1 mon.magna102@0(leader).log v20995 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20995)
>2018-07-11 11:15:54.145164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:15:54.681322 7f29149e4700  1 mon.magna102@0(leader).log v20996 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 20996)
>2018-07-11 11:15:55.306715 7f29149e4700  1 mon.magna102@0(leader).osd e566 e566: 12 total, 11 up, 12 in
>2018-07-11 11:15:55.340588 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e566: 12 total, 11 up, 12 in
>2018-07-11 11:15:55.450425 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:15:55.450448 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3456789301' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:15:56.373631 7f29149e4700  1 mon.magna102@0(leader).log v20997 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20997)
>2018-07-11 11:15:56.440565 7f29149e4700  1 mon.magna102@0(leader).osd e567 e567: 12 total, 11 up, 12 in
>2018-07-11 11:15:56.474370 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e567: 12 total, 11 up, 12 in
>2018-07-11 11:15:57.482361 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:15:57.482389 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3278519120' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:15:57.515827 7f29149e4700  1 mon.magna102@0(leader).log v20998 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20998)
>2018-07-11 11:15:58.515843 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:15:58.515892 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 95234/24217500 objects degraded (0.393%), 4 pgs degraded (PG_DEGRADED)
>2018-07-11 11:15:58.649471 7f29149e4700  1 mon.magna102@0(leader).log v20999 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 20999)
>2018-07-11 11:15:59.716389 7f29149e4700  1 mon.magna102@0(leader).log v21000 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21000)
>2018-07-11 11:16:00.187676 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:00.187699 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1904920270' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:00.749837 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 11:16:00.783226 7f29149e4700  1 mon.magna102@0(leader).log v21001 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21001)
>2018-07-11 11:16:01.618216 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:01.618238 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3427839849' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:01.850346 7f29149e4700  1 mon.magna102@0(leader).log v21002 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21002)
>2018-07-11 11:16:02.917860 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:16:02.967541 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:02.967564 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2918643612' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:03.983953 7f29149e4700  1 mon.magna102@0(leader).log v21003 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21003)
>2018-07-11 11:16:04.145643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 25 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:16:04.145697 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 800366/24217500 objects degraded (3.305%), 31 pgs degraded (PG_DEGRADED)
>2018-07-11 11:16:04.299539 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:04.299564 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/553887767' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:05.050714 7f29149e4700  1 mon.magna102@0(leader).log v21004 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21004)
>2018-07-11 11:16:05.716082 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:05.716105 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2241156526' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:06.251330 7f29149e4700  1 mon.magna102@0(leader).log v21005 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21005)
>2018-07-11 11:16:06.999919 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:06.999941 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3921789812' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:07.318197 7f29149e4700  1 mon.magna102@0(leader).log v21006 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21006)
>2018-07-11 11:16:08.247739 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:08.247763 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/896419559' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:08.385015 7f29149e4700  1 mon.magna102@0(leader).log v21007 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21007)
>2018-07-11 11:16:09.500593 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:09.500616 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/822361517' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:09.618734 7f29149e4700  1 mon.magna102@0(leader).log v21008 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21008)
>2018-07-11 11:16:10.842650 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:10.842673 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1009704837' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:11.763011 7f29149e4700  1 mon.magna102@0(leader).log v21009 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21009)
>2018-07-11 11:16:16.129144 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:16.279373 7f29149e4700  1 mon.magna102@0(leader).log v21010 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21010)
>2018-07-11 11:16:24.171363 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 43 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:24.288398 7f29149e4700  1 mon.magna102@0(leader).log v21011 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21011)
>2018-07-11 11:16:36.140279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 49 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:36.289469 7f29149e4700  1 mon.magna102@0(leader).log v21012 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21012)
>2018-07-11 11:16:42.143406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 53 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:42.298417 7f29149e4700  1 mon.magna102@0(leader).log v21013 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21013)
>2018-07-11 11:16:49.181944 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 57 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:49.290648 7f29149e4700  1 mon.magna102@0(leader).log v21014 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21014)
>2018-07-11 11:16:52.148926 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 3 pgs inactive, 25 pgs peering, 5 pgs stale (PG_AVAILABILITY)
>2018-07-11 11:16:52.299494 7f29149e4700  1 mon.magna102@0(leader).log v21015 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21015)
>2018-07-11 11:16:54.182832 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 59 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:16:54.301076 7f29149e4700  1 mon.magna102@0(leader).log v21016 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21016)
>2018-07-11 11:16:54.514890 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:54.514913 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2394805218' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:55.367919 7f29149e4700  1 mon.magna102@0(leader).log v21017 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21017)
>2018-07-11 11:16:55.773071 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:55.773094 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/248250883' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:56.301508 7f29149e4700  1 mon.magna102@0(leader).osd e568 e568: 12 total, 11 up, 12 in
>2018-07-11 11:16:56.335213 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e568: 12 total, 11 up, 12 in
>2018-07-11 11:16:56.501930 7f29149e4700  1 mon.magna102@0(leader).log v21018 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21018)
>2018-07-11 11:16:57.077557 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:57.077580 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2727478493' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:57.577243 7f29149e4700  1 mon.magna102@0(leader).log v21019 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21019)
>2018-07-11 11:16:58.403864 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:16:58.403887 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3666086106' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:16:58.577319 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 24 pgs inactive, 24 pgs peering, 5 pgs stale (PG_AVAILABILITY)
>2018-07-11 11:16:58.577373 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 868775/24217500 objects degraded (3.587%), 32 pgs degraded, 8 pgs undersized (PG_DEGRADED)
>2018-07-11 11:16:58.719208 7f29149e4700  1 mon.magna102@0(leader).log v21020 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21020)
>2018-07-11 11:16:59.183054 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 57 slow requests are blocked > 32 sec. Implicated osds 6,8 (REQUEST_SLOW)
>2018-07-11 11:16:59.270098 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10547. Immutable memtables: 0.
>
>2018-07-11 11:16:59.270175 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 947] Syncing log #10544
>2018-07-11 11:16:59.286226 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.270164) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:16:59.286236 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 947] Flushing memtable with next log file: 10547
>
>2018-07-11 11:16:59.286264 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819286255, "job": 947, "event": "flush_started", "num_memtables": 1, "num_entries": 1906, "num_deletes": 502, "memory_usage": 3436816}
>2018-07-11 11:16:59.286267 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 947] Level-0 flush table #10548: started
>2018-07-11 11:16:59.386947 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819386932, "cf_name": "default", "job": 947, "event": "table_file_creation", "file_number": 10548, "file_size": 3178122, "table_properties": {"data_size": 3153130, "index_size": 13039, "filter_size": 11028, "raw_key_size": 19547, "raw_average_key_size": 20, "raw_value_size": 3130841, "raw_average_value_size": 3316, "num_data_blocks": 393, "num_entries": 944, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "502", "kMergeOperands": "0"}}
>2018-07-11 11:16:59.386971 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 947] Level-0 flush table #10548: 3178122 bytes OK
>2018-07-11 11:16:59.478712 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.414178) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10548 started
>2018-07-11 11:16:59.478721 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.478673) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10548: memtable #1 done
>2018-07-11 11:16:59.478723 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.478688) EVENT_LOG_v1 {"time_micros": 1531307819478682, "job": 947, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:16:59.478725 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.478702) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:16:59.478744 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 947] Try to delete WAL files size 3397616, prev total WAL file size 3410947, number of live WAL files 2.
>
>2018-07-11 11:16:59.479669 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:16:59.479861 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 11:16:59.479867 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 69846.3 total, 731.4 interval
>Cumulative writes: 94K writes, 445K keys, 94K commit groups, 1.0 writes per commit group, ingest: 0.74 GB, 0.01 MB/s
>Cumulative WAL: 94K writes, 94K syncs, 1.00 writes per sync, written: 0.74 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 894 writes, 4448 keys, 894 commit groups, 1.0 writes per commit group, ingest: 6.96 MB, 0.01 MB/s
>Interval WAL: 894 writes, 894 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.03 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.1        38       474    0.081       0      0
>  L1      1/1   15.78 MB   0.0      7.5     0.6      6.8       6.8      0.0       0.0  10.7     72.6     66.4       105       472    0.223   1690K   202K
> Sum      2/2   18.81 MB   0.0      7.5     0.6      6.8       7.5      0.6       0.0  11.7     53.2     53.3       144       946    0.152   1690K   202K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0   9.1     53.6     54.5         1         6    0.164     11K   1419
>Uptime(secs): 69846.3 total, 69846.3 interval
>Flush(GB): cumulative 0.641, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.47 GB write, 0.11 MB/s write, 7.47 GB read, 0.11 MB/s read, 143.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 1.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.03 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.1        38       474    0.081       0      0
>  L1      1/1   15.78 MB   0.0      7.5     0.6      6.8       6.8      0.0       0.0  10.7     72.6     66.4       105       472    0.223   1690K   202K
> Sum      2/2   18.81 MB   0.0      7.5     0.6      6.8       7.5      0.6       0.0  11.7     53.2     53.3       144       946    0.152   1690K   202K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 69846.3 total, 69846.3 interval
>Flush(GB): cumulative 0.641, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.47 GB write, 0.11 MB/s write, 7.47 GB read, 0.11 MB/s read, 143.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 11:16:59.479889 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.479882) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 11:16:59.479893 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 948] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:16:59.479899 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 947 Base level 0, inputs: [10548(3103KB)], [10546(15MB)]
>
>2018-07-11 11:16:59.479911 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819479902, "job": 948, "event": "compaction_started", "files_L0": [10548], "files_L1": [10546], "score": -1, "input_data_size": 19724477}
>2018-07-11 11:16:59.679406 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 948] Generated table #10549: 3451 keys, 17350546 bytes
>2018-07-11 11:16:59.679451 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819679435, "cf_name": "default", "job": 948, "event": "table_file_creation", "file_number": 10549, "file_size": 17350546, "table_properties": {"data_size": 17218256, "index_size": 75746, "filter_size": 55617, "raw_key_size": 69817, "raw_average_key_size": 20, "raw_value_size": 17116076, "raw_average_value_size": 4959, "num_data_blocks": 2218, "num_entries": 3451, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:16:59.712589 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 948] Compacted 1@0 + 1@1 files to L1 => 17350546 bytes
>2018-07-11 11:16:59.755970 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.755888) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 84.8 rd, 74.6 wr, level 1, files in(1, 1) out(1) MB in(3.0, 15.8) out(16.5), read-write-amplify(11.7) write-amplify(5.5) OK, records in: 4467, records dropped: 1016
>
>2018-07-11 11:16:59.755980 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.755912) EVENT_LOG_v1 {"time_micros": 1531307819755902, "job": 948, "event": "compaction_finished", "compaction_time_micros": 232661, "output_level": 1, "num_output_files": 1, "total_output_size": 17350546, "num_input_records": 4467, "num_output_records": 3451, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:16:59.756734 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819756729, "job": 948, "event": "table_file_deletion", "file_number": 10548}
>2018-07-11 11:16:59.759881 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819759878, "job": 948, "event": "table_file_deletion", "file_number": 10546}
>2018-07-11 11:16:59.779712 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10550. Immutable memtables: 0.
>
>2018-07-11 11:16:59.779753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 949] Syncing log #10547
>2018-07-11 11:16:59.796520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.779742) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:16:59.796531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 949] Flushing memtable with next log file: 10550
>
>2018-07-11 11:16:59.796557 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819796550, "job": 949, "event": "flush_started", "num_memtables": 1, "num_entries": 258, "num_deletes": 250, "memory_usage": 30680}
>2018-07-11 11:16:59.796560 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 949] Level-0 flush table #10551: started
>2018-07-11 11:16:59.855515 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819855502, "cf_name": "default", "job": 949, "event": "table_file_creation", "file_number": 10551, "file_size": 24505, "table_properties": {"data_size": 22800, "index_size": 85, "filter_size": 702, "raw_key_size": 5646, "raw_average_key_size": 22, "raw_value_size": 19331, "raw_average_value_size": 75, "num_data_blocks": 3, "num_entries": 256, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:16:59.855526 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 949] Level-0 flush table #10551: 24505 bytes OK
>2018-07-11 11:16:59.855533 7f29149e4700  1 mon.magna102@0(leader).log v21021 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 21021)
>2018-07-11 11:16:59.905459 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.871988) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10551 started
>2018-07-11 11:16:59.905468 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.905424) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10551: memtable #1 done
>2018-07-11 11:16:59.905471 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.905439) EVENT_LOG_v1 {"time_micros": 1531307819905433, "job": 949, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:16:59.905474 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.905451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:16:59.905479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 949] Try to delete WAL files size 23538, prev total WAL file size 33607, number of live WAL files 2.
>
>2018-07-11 11:16:59.905577 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:16:59.905762 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10552. Immutable memtables: 0.
>
>2018-07-11 11:16:59.905790 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 950] Syncing log #10550
>2018-07-11 11:16:59.922526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.905784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:16:59.922537 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 950] Flushing memtable with next log file: 10552
>
>2018-07-11 11:16:59.922561 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819922542, "job": 950, "event": "flush_started", "num_memtables": 1, "num_entries": 5, "num_deletes": 0, "memory_usage": 12312}
>2018-07-11 11:16:59.922564 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 950] Level-0 flush table #10553: started
>2018-07-11 11:16:59.964536 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307819964522, "cf_name": "default", "job": 950, "event": "table_file_creation", "file_number": 10553, "file_size": 11122, "table_properties": {"data_size": 10113, "index_size": 57, "filter_size": 43, "raw_key_size": 120, "raw_average_key_size": 24, "raw_value_size": 9960, "raw_average_value_size": 1992, "num_data_blocks": 2, "num_entries": 5, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:16:59.964547 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 950] Level-0 flush table #10553: 11122 bytes OK
>2018-07-11 11:17:00.014727 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:16:59.981290) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10553 started
>2018-07-11 11:17:00.014735 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.014695) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10553: memtable #1 done
>2018-07-11 11:17:00.014737 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.014708) EVENT_LOG_v1 {"time_micros": 1531307820014703, "job": 950, "event": "flush_finished", "lsm_state": [2, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:17:00.014752 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.014719) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 1 0 0 0 0 0] max score 0.50
>
>2018-07-11 11:17:00.014757 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 950] Try to delete WAL files size 10069, prev total WAL file size 10069, number of live WAL files 2.
>
>2018-07-11 11:17:00.014852 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:17:00.014902 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.014896) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:17:00.014911 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 951] Compacting 2@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:17:00.014917 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 950 Base level 0, inputs: [10553(10KB) 10551(23KB)], [10549(16MB)]
>
>2018-07-11 11:17:00.014931 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307820014921, "job": 951, "event": "compaction_started", "files_L0": [10553, 10551], "files_L1": [10549], "score": -1, "input_data_size": 17386173}
>2018-07-11 11:17:00.198680 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 951] Generated table #10554: 3205 keys, 15671636 bytes
>2018-07-11 11:17:00.198720 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307820198694, "cf_name": "default", "job": 951, "event": "table_file_creation", "file_number": 10554, "file_size": 15671636, "table_properties": {"data_size": 15553440, "index_size": 67124, "filter_size": 50145, "raw_key_size": 64398, "raw_average_key_size": 20, "raw_value_size": 15460883, "raw_average_value_size": 4823, "num_data_blocks": 1971, "num_entries": 3205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:17:00.231930 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 951] Compacted 2@0 + 1@1 files to L1 => 15671636 bytes
>2018-07-11 11:17:00.265461 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.265384) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.1 rd, 72.2 wr, level 1, files in(2, 1) out(1) MB in(0.0, 16.5) out(14.9), read-write-amplify(927.9) write-amplify(439.9) OK, records in: 3712, records dropped: 507
>
>2018-07-11 11:17:00.265469 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:17:00.265406) EVENT_LOG_v1 {"time_micros": 1531307820265396, "job": 951, "event": "compaction_finished", "compaction_time_micros": 216984, "output_level": 1, "num_output_files": 1, "total_output_size": 15671636, "num_input_records": 3712, "num_output_records": 3205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:17:00.265563 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307820265559, "job": 951, "event": "table_file_deletion", "file_number": 10553}
>2018-07-11 11:17:00.265610 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307820265607, "job": 951, "event": "table_file_deletion", "file_number": 10551}
>2018-07-11 11:17:00.268913 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307820268910, "job": 951, "event": "table_file_deletion", "file_number": 10549}
>2018-07-11 11:17:00.269025 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:17:00.855416 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 11:17:01.007877 7f29149e4700  1 mon.magna102@0(leader).log v21022 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21022)
>2018-07-11 11:17:02.918046 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 11:17:04.124982 7f29149e4700  1 mon.magna102@0(leader).osd e569 e569: 12 total, 11 up, 12 in
>2018-07-11 11:17:04.158719 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e569: 12 total, 11 up, 12 in
>2018-07-11 11:17:04.230347 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 21 pgs inactive, 22 pgs peering, 2 pgs stale (PG_AVAILABILITY)
>2018-07-11 11:17:04.230391 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1307302/24217500 objects degraded (5.398%), 40 pgs degraded, 36 pgs undersized (PG_DEGRADED)
>2018-07-11 11:17:04.230410 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:17:05.201728 7f29149e4700  1 mon.magna102@0(leader).log v21023 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 21023)
>2018-07-11 11:17:06.133873 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:06.133895 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4081839988' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:06.285196 7f29149e4700  1 mon.magna102@0(leader).log v21024 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21024)
>2018-07-11 11:17:07.525009 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:07.525032 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2110532384' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:07.660583 7f29149e4700  1 mon.magna102@0(leader).log v21025 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21025)
>2018-07-11 11:17:08.919623 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:08.919647 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3680316062' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:09.230662 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 21 pgs inactive, 16 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:17:09.230715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1526626/24217500 objects degraded (6.304%), 48 pgs degraded, 36 pgs undersized (PG_DEGRADED)
>2018-07-11 11:17:09.827560 7f29149e4700  1 mon.magna102@0(leader).log v21026 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21026)
>2018-07-11 11:17:10.099352 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:10.099376 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1470940405' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:10.827534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:17:10.994598 7f29149e4700  1 mon.magna102@0(leader).log v21027 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21027)
>2018-07-11 11:17:11.413155 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:11.413188 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2650759475' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:12.078208 7f29149e4700  1 mon.magna102@0(leader).log v21028 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21028)
>2018-07-11 11:17:12.737475 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:12.737498 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3821802769' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:13.245148 7f29149e4700  1 mon.magna102@0(leader).log v21029 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21029)
>2018-07-11 11:17:14.131343 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:14.131366 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/798386051' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:14.231006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 21 pgs inactive, 3 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:17:14.231060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1927663/24217500 objects degraded (7.960%), 59 pgs degraded, 36 pgs undersized (PG_DEGRADED)
>2018-07-11 11:17:14.328639 7f29149e4700  1 mon.magna102@0(leader).log v21030 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21030)
>2018-07-11 11:17:15.417572 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:15.417594 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3341288573' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:15.545660 7f29149e4700  1 mon.magna102@0(leader).log v21031 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21031)
>2018-07-11 11:17:16.545665 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:17:16.706706 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:16.706729 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1844241779' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:16.721148 7f29149e4700  1 mon.magna102@0(leader).log v21032 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21032)
>2018-07-11 11:17:17.804558 7f29149e4700  1 mon.magna102@0(leader).log v21033 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21033)
>2018-07-11 11:17:24.231540 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 24 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:17:24.363842 7f29149e4700  1 mon.magna102@0(leader).log v21034 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21034)
>2018-07-11 11:17:29.231794 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 56 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:29.330957 7f29149e4700  1 mon.magna102@0(leader).log v21035 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21035)
>2018-07-11 11:17:36.178368 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 62 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:36.348492 7f29149e4700  1 mon.magna102@0(leader).log v21036 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21036)
>2018-07-11 11:17:37.386946 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:37.386970 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4055135255' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:37.515325 7f29149e4700  1 mon.magna102@0(leader).log v21037 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21037)
>2018-07-11 11:17:38.677689 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:38.677712 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1362222652' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:39.682277 7f29149e4700  1 mon.magna102@0(leader).log v21038 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21038)
>2018-07-11 11:17:39.961218 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:17:39.961242 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1969453449' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:17:40.849344 7f29149e4700  1 mon.magna102@0(leader).log v21039 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21039)
>2018-07-11 11:17:44.232487 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 64 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:44.359539 7f29149e4700  1 mon.magna102@0(leader).log v21040 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21040)
>2018-07-11 11:17:49.232759 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 66 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:49.318338 7f29149e4700  1 mon.magna102@0(leader).log v21041 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21041)
>2018-07-11 11:17:54.233009 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 68 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:54.352363 7f29149e4700  1 mon.magna102@0(leader).log v21042 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21042)
>2018-07-11 11:17:58.192381 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 22 pgs inactive, 3 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:17:58.192429 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1927663/24217500 objects degraded (7.960%), 59 pgs degraded, 42 pgs undersized (PG_DEGRADED)
>2018-07-11 11:17:58.352907 7f29149e4700  1 mon.magna102@0(leader).log v21043 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21043)
>2018-07-11 11:17:59.233264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 70 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:17:59.419735 7f29149e4700  1 mon.magna102@0(leader).log v21044 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21044)
>2018-07-11 11:18:00.553617 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10555. Immutable memtables: 0.
>
>2018-07-11 11:18:00.553664 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 952] Syncing log #10552
>2018-07-11 11:18:00.570185 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.553654) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:18:00.570197 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 952] Flushing memtable with next log file: 10555
>
>2018-07-11 11:18:00.570211 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880570203, "job": 952, "event": "flush_started", "num_memtables": 1, "num_entries": 667, "num_deletes": 251, "memory_usage": 1461728}
>2018-07-11 11:18:00.570214 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 952] Level-0 flush table #10556: started
>2018-07-11 11:18:00.620717 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880620703, "cf_name": "default", "job": 952, "event": "table_file_creation", "file_number": 10556, "file_size": 1200089, "table_properties": {"data_size": 1191390, "index_size": 3743, "filter_size": 4037, "raw_key_size": 8121, "raw_average_key_size": 20, "raw_value_size": 1183203, "raw_average_value_size": 2972, "num_data_blocks": 114, "num_entries": 398, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:18:00.620728 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 952] Level-0 flush table #10556: 1200089 bytes OK
>2018-07-11 11:18:00.670622 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.637272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10556 started
>2018-07-11 11:18:00.670630 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.670587) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10556: memtable #1 done
>2018-07-11 11:18:00.670633 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.670601) EVENT_LOG_v1 {"time_micros": 1531307880670595, "job": 952, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:18:00.670635 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.670612) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:18:00.670642 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 952] Try to delete WAL files size 1446596, prev total WAL file size 1446596, number of live WAL files 2.
>
>2018-07-11 11:18:00.671042 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:18:00.671096 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.671088) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:18:00.671105 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 953] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:18:00.671111 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 952 Base level 0, inputs: [10556(1171KB)], [10554(14MB)]
>
>2018-07-11 11:18:00.671123 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880671114, "job": 953, "event": "compaction_started", "files_L0": [10556], "files_L1": [10554], "score": -1, "input_data_size": 16871725}
>2018-07-11 11:18:00.846473 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 953] Generated table #10557: 3089 keys, 14888204 bytes
>2018-07-11 11:18:00.846511 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880846494, "cf_name": "default", "job": 953, "event": "table_file_creation", "file_number": 10557, "file_size": 14888204, "table_properties": {"data_size": 14777132, "index_size": 62715, "filter_size": 47430, "raw_key_size": 62164, "raw_average_key_size": 20, "raw_value_size": 14689263, "raw_average_value_size": 4755, "num_data_blocks": 1833, "num_entries": 3089, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:18:00.879543 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 953] Compacted 1@0 + 1@1 files to L1 => 14888204 bytes
>2018-07-11 11:18:00.913175 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.913104) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.0 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.9) out(14.2), read-write-amplify(26.5) write-amplify(12.4) OK, records in: 3603, records dropped: 514
>
>2018-07-11 11:18:00.913194 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:18:00.913125) EVENT_LOG_v1 {"time_micros": 1531307880913116, "job": 953, "event": "compaction_finished", "compaction_time_micros": 208406, "output_level": 1, "num_output_files": 1, "total_output_size": 14888204, "num_input_records": 3603, "num_output_records": 3089, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:18:00.913473 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880913469, "job": 953, "event": "table_file_deletion", "file_number": 10556}
>2018-07-11 11:18:00.916401 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531307880916398, "job": 953, "event": "table_file_deletion", "file_number": 10554}
>2018-07-11 11:18:02.918238 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:18:04.233525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1927663/24217500 objects degraded (7.960%), 59 pgs degraded, 45 pgs undersized (PG_DEGRADED)
>2018-07-11 11:18:04.233573 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 76 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:04.355861 7f29149e4700  1 mon.magna102@0(leader).log v21045 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21045)
>2018-07-11 11:18:06.197708 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 3 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:18:06.347668 7f29149e4700  1 mon.magna102@0(leader).log v21046 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21046)
>2018-07-11 11:18:09.233827 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1927663/24217500 objects degraded (7.960%), 59 pgs degraded, 67 pgs undersized (PG_DEGRADED)
>2018-07-11 11:18:09.331545 7f29149e4700  1 mon.magna102@0(leader).log v21047 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21047)
>2018-07-11 11:18:10.331566 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 80 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:10.490243 7f29149e4700  1 mon.magna102@0(leader).log v21048 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21048)
>2018-07-11 11:18:14.812430 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]} v 0) v1
>2018-07-11 11:18:14.812469 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/17638' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["10"]}]: dispatch
>2018-07-11 11:18:14.813133 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]} v 0) v1
>2018-07-11 11:18:14.813168 7f29191ed700  0 log_channel(audit) log [INF] : from='osd.10 10.8.128.107:6800/17638' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.9096, "args": ["host=magna107", "root=default"]}]: dispatch
>2018-07-11 11:18:14.813263 7f29191ed700  0 mon.magna102@0(leader).osd e569 create-or-move crush item name 'osd.10' initial_weight 0.9096 at location {host=magna107,root=default}
>2018-07-11 11:18:15.340677 7f29149e4700  1 mon.magna102@0(leader).log v21049 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21049)
>2018-07-11 11:18:15.415948 7f29149e4700  1 mon.magna102@0(leader).osd e570 e570: 12 total, 11 up, 12 in
>2018-07-11 11:18:15.449709 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e570: 12 total, 11 up, 12 in
>2018-07-11 11:18:16.449452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 88 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:16.450277 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down)
>2018-07-11 11:18:16.482874 7f29149e4700  1 mon.magna102@0(leader).log v21050 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21050)
>2018-07-11 11:18:16.559503 7f29149e4700  1 mon.magna102@0(leader).osd e571 e571: 12 total, 12 up, 12 in
>2018-07-11 11:18:16.593109 7f29149e4700  0 log_channel(cluster) log [INF] : osd.10 10.8.128.107:6800/17638 boot
>2018-07-11 11:18:16.593240 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e571: 12 total, 12 up, 12 in
>2018-07-11 11:18:16.593430 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd metadata", "id": 10} v 0) v1
>2018-07-11 11:18:16.593467 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.24106 10.8.128.102:0/2030830768' entity='mgr.magna102' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch
>2018-07-11 11:18:17.559694 7f29149e4700  1 mon.magna102@0(leader).log v21051 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 21051)
>2018-07-11 11:18:17.634839 7f29149e4700  1 mon.magna102@0(leader).osd e572 e572: 12 total, 12 up, 12 in
>2018-07-11 11:18:17.668769 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e572: 12 total, 12 up, 12 in
>2018-07-11 11:18:18.676752 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 6 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:18:18.676791 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 1736517/24217500 objects degraded (7.171%), 54 pgs degraded, 60 pgs undersized (PG_DEGRADED)
>2018-07-11 11:18:18.718512 7f29149e4700  1 mon.magna102@0(leader).log v21052 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21052)
>2018-07-11 11:18:19.802344 7f29149e4700  1 mon.magna102@0(leader).log v21053 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21053)
>2018-07-11 11:18:20.802314 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: OBJECT_MISPLACED (was: 2691/24217500 objects misplaced (0.011%))
>2018-07-11 11:18:20.977677 7f29149e4700  1 mon.magna102@0(leader).log v21054 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21054)
>2018-07-11 11:18:22.904894 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:22.904916 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2109844782' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:23.369664 7f29149e4700  1 mon.magna102@0(leader).log v21055 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21055)
>2018-07-11 11:18:24.239820 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:24.239843 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4177449866' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:24.240712 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 20 pgs inactive, 27 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:18:24.240758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 139022/24217500 objects degraded (0.574%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED)
>2018-07-11 11:18:24.240777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 92 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:24.453279 7f29149e4700  1 mon.magna102@0(leader).log v21056 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 21056)
>2018-07-11 11:18:26.247127 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:26.247150 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3608804792' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:26.378736 7f29149e4700  1 mon.magna102@0(leader).log v21057 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21057)
>2018-07-11 11:18:27.695268 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:27.695291 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1427522623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:27.828907 7f29149e4700  1 mon.magna102@0(leader).log v21058 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21058)
>2018-07-11 11:18:28.912352 7f29149e4700  1 mon.magna102@0(leader).log v21059 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21059)
>2018-07-11 11:18:29.069242 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:29.069264 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3974771243' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:30.079426 7f29149e4700  1 mon.magna102@0(leader).log v21060 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21060)
>2018-07-11 11:18:30.354487 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:30.354511 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2190509692' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:31.121264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 94 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:31.163012 7f29149e4700  1 mon.magna102@0(leader).log v21061 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21061)
>2018-07-11 11:18:31.661092 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:31.661115 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1015157725' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:32.280134 7f29149e4700  1 mon.magna102@0(leader).log v21062 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21062)
>2018-07-11 11:18:33.029709 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:33.029732 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2263529675' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:33.438821 7f29149e4700  1 mon.magna102@0(leader).log v21063 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21063)
>2018-07-11 11:18:33.930722 7f29149e4700  1 mon.magna102@0(leader).osd e573 e573: 12 total, 12 up, 12 in
>2018-07-11 11:18:33.972937 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e573: 12 total, 12 up, 12 in
>2018-07-11 11:18:35.072801 7f29149e4700  1 mon.magna102@0(leader).osd e574 e574: 12 total, 12 up, 12 in
>2018-07-11 11:18:35.106167 7f29149e4700  1 mon.magna102@0(leader).log v21064 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21064)
>2018-07-11 11:18:35.106508 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e574: 12 total, 12 up, 12 in
>2018-07-11 11:18:36.139770 7f29149e4700  1 mon.magna102@0(leader).log v21065 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21065)
>2018-07-11 11:18:37.139819 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 20 pgs inactive, 25 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:18:37.139865 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 96 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:18:37.139885 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 139022/24217500 objects degraded (0.574%), 3 pgs degraded, 3 pgs undersized)
>2018-07-11 11:18:37.315208 7f29149e4700  1 mon.magna102@0(leader).log v21066 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 3 entries (version 21066)
>2018-07-11 11:18:39.060924 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:39.060945 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/2610814777' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:39.482138 7f29149e4700  1 mon.magna102@0(leader).log v21067 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21067)
>2018-07-11 11:18:40.482207 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2691/24217500 objects misplaced (0.011%) (OBJECT_MISPLACED)
>2018-07-11 11:18:40.657535 7f29149e4700  1 mon.magna102@0(leader).log v21068 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21068)
>2018-07-11 11:18:42.219084 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 94 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:18:42.382873 7f29149e4700  1 mon.magna102@0(leader).log v21069 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21069)
>2018-07-11 11:18:42.673589 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:42.673608 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/1781633321' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:43.459350 7f29149e4700  1 mon.magna102@0(leader).log v21070 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21070)
>2018-07-11 11:18:46.221305 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 20 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:18:46.368228 7f29149e4700  1 mon.magna102@0(leader).log v21071 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21071)
>2018-07-11 11:18:49.241763 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 34 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:18:49.360218 7f29149e4700  1 mon.magna102@0(leader).log v21072 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21072)
>2018-07-11 11:18:51.728327 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:18:51.728345 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/319428194' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:18:51.729583 7f29191ed700  1 mon.magna102@0(leader).log v21072 check_sub sending message to client.? 10.8.128.107:0/319428194 with 1 entries (version 21072)
>2018-07-11 11:18:51.843831 7f29149e4700  1 mon.magna102@0(leader).log v21073 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21073)
>2018-07-11 11:18:51.843851 7f29149e4700  1 mon.magna102@0(leader).log v21073 check_sub sending message to client.? 10.8.128.107:0/319428194 with 0 entries (version 21073)
>2018-07-11 11:18:56.226560 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 51 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:18:56.377925 7f29149e4700  1 mon.magna102@0(leader).log v21074 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21074)
>2018-07-11 11:19:02.229632 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 6,9 (REQUEST_SLOW)
>2018-07-11 11:19:02.378542 7f29149e4700  1 mon.magna102@0(leader).log v21075 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21075)
>2018-07-11 11:19:02.918394 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:19:03.886247 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:19:03.886265 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3120705353' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:19:03.887705 7f29191ed700  1 mon.magna102@0(leader).log v21075 check_sub sending message to client.? 10.8.128.107:0/3120705353 with 1 entries (version 21075)
>2018-07-11 11:19:04.003650 7f29149e4700  1 mon.magna102@0(leader).log v21076 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21076)
>2018-07-11 11:19:04.003669 7f29149e4700  1 mon.magna102@0(leader).log v21076 check_sub sending message to client.? 10.8.128.107:0/3120705353 with 0 entries (version 21076)
>2018-07-11 11:19:08.024970 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:19:08.024989 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/771314125' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:19:08.179238 7f29149e4700  1 mon.magna102@0(leader).log v21077 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21077)
>2018-07-11 11:19:09.261358 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 63 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:09.526221 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:19:09.526241 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/1328955691' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:19:10.413150 7f29149e4700  1 mon.magna102@0(leader).log v21078 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21078)
>2018-07-11 11:19:10.890100 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:19:10.890119 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/2315222673' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:19:11.504963 7f29149e4700  1 mon.magna102@0(leader).log v21079 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21079)
>2018-07-11 11:19:12.224843 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:19:12.224862 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/4059455239' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:19:12.638637 7f29149e4700  1 mon.magna102@0(leader).log v21080 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21080)
>2018-07-11 11:19:14.280331 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 65 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:14.397382 7f29149e4700  1 mon.magna102@0(leader).log v21081 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21081)
>2018-07-11 11:19:18.238123 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 25 pgs inactive, 24 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:19:18.389602 7f29149e4700  1 mon.magna102@0(leader).log v21082 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21082)
>2018-07-11 11:19:19.280562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 67 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:19.456429 7f29149e4700  1 mon.magna102@0(leader).log v21083 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21083)
>2018-07-11 11:19:30.244725 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 69 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:30.400388 7f29149e4700  1 mon.magna102@0(leader).log v21084 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21084)
>2018-07-11 11:19:36.247968 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 77 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:36.417655 7f29149e4700  1 mon.magna102@0(leader).log v21085 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21085)
>2018-07-11 11:19:44.252213 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 60232/24217500 objects degraded (0.249%), 1 pg undersized (PG_DEGRADED)
>2018-07-11 11:19:44.293025 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 85 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:44.426753 7f29149e4700  1 mon.magna102@0(leader).log v21086 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21086)
>2018-07-11 11:19:49.293257 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 91 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:49.418825 7f29149e4700  1 mon.magna102@0(leader).log v21087 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21087)
>2018-07-11 11:19:54.294199 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 95 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:54.427844 7f29149e4700  1 mon.magna102@0(leader).log v21088 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21088)
>2018-07-11 11:19:59.294421 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 97 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:19:59.461821 7f29149e4700  1 mon.magna102@0(leader).log v21089 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21089)
>2018-07-11 11:20:02.799835 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:02.799857 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/2041636631' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:02.918530 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:20:03.429330 7f29149e4700  1 mon.magna102@0(leader).log v21090 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21090)
>2018-07-11 11:20:04.336651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 103 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:20:04.400399 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:04.400418 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3826810791' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:04.512875 7f29149e4700  1 mon.magna102@0(leader).log v21091 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21091)
>2018-07-11 11:20:05.723150 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:05.723169 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/1629450644' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:05.854863 7f29149e4700  1 mon.magna102@0(leader).log v21092 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21092)
>2018-07-11 11:20:07.118831 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:07.118850 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/404007710' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:08.021890 7f29149e4700  1 mon.magna102@0(leader).log v21093 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21093)
>2018-07-11 11:20:08.578303 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:08.578322 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/818322724' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:09.180575 7f29149e4700  1 mon.magna102@0(leader).log v21094 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21094)
>2018-07-11 11:20:09.336901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 107 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:20:10.004194 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:10.004213 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3494987527' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:10.247466 7f29149e4700  1 mon.magna102@0(leader).log v21095 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21095)
>2018-07-11 11:20:14.337148 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 109 slow requests are blocked > 32 sec. Implicated osds 6,8,9 (REQUEST_SLOW)
>2018-07-11 11:20:14.456498 7f29149e4700  1 mon.magna102@0(leader).log v21096 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21096)
>2018-07-11 11:20:19.249947 7f29149e4700  1 mon.magna102@0(leader).osd e575 e575: 12 total, 12 up, 12 in
>2018-07-11 11:20:19.283632 7f29149e4700  0 log_channel(cluster) log [DBG] : osdmap e575: 12 total, 12 up, 12 in
>2018-07-11 11:20:19.337408 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 107 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:20:20.249910 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 16 pgs inactive, 16 pgs peering (PG_AVAILABILITY)
>2018-07-11 11:20:20.249956 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60232/24217500 objects degraded (0.249%), 1 pg undersized)
>2018-07-11 11:20:20.383744 7f29149e4700  1 mon.magna102@0(leader).log v21097 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21097)
>2018-07-11 11:20:21.450456 7f29149e4700  1 mon.magna102@0(leader).log v21098 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21098)
>2018-07-11 11:20:24.337640 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 109 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 11:20:24.459447 7f29149e4700  1 mon.magna102@0(leader).log v21099 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21099)
>2018-07-11 11:20:26.274454 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 109 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 11:20:26.426395 7f29149e4700  1 mon.magna102@0(leader).log v21100 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21100)
>2018-07-11 11:20:29.337873 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)
>2018-07-11 11:20:29.460172 7f29149e4700  1 mon.magna102@0(leader).log v21101 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21101)
>2018-07-11 11:20:42.281609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:20:42.436601 7f29149e4700  1 mon.magna102@0(leader).log v21102 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21102)
>2018-07-11 11:20:56.286713 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:56.286736 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3476389979' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:56.421051 7f29149e4700  1 mon.magna102@0(leader).log v21103 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21103)
>2018-07-11 11:20:58.173579 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:20:58.173602 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2389815221' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:20:58.296236 7f29149e4700  1 mon.magna102@0(leader).log v21104 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21104)
>2018-07-11 11:21:00.356080 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:00.356107 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3203253864' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:00.357411 7f29191ed700  1 mon.magna102@0(leader).log v21104 check_sub sending message to client.? 10.8.128.102:0/3203253864 with 1 entries (version 21104)
>2018-07-11 11:21:00.480100 7f29149e4700  1 mon.magna102@0(leader).log v21105 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21105)
>2018-07-11 11:21:00.480133 7f29149e4700  1 mon.magna102@0(leader).log v21105 check_sub sending message to client.? 10.8.128.102:0/3203253864 with 0 entries (version 21105)
>2018-07-11 11:21:02.918710 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:21:04.006508 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:04.006531 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3289487404' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:04.130521 7f29149e4700  1 mon.magna102@0(leader).log v21106 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21106)
>2018-07-11 11:21:06.255969 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:06.255992 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3971184660' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:06.457284 7f29149e4700  1 mon.magna102@0(leader).log v21107 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21107)
>2018-07-11 11:21:07.589711 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:07.589735 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3968477164' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:07.724121 7f29149e4700  1 mon.magna102@0(leader).log v21108 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21108)
>2018-07-11 11:21:21.433638 7f29149e4700  1 mon.magna102@0(leader).log v21109 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21109)
>2018-07-11 11:21:25.677967 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:25.677991 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1400224748' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:25.792430 7f29149e4700  1 mon.magna102@0(leader).log v21110 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21110)
>2018-07-11 11:21:27.388127 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:27.388152 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1543572304' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:27.926219 7f29149e4700  1 mon.magna102@0(leader).log v21111 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21111)
>2018-07-11 11:21:28.812897 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:28.812924 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4257723194' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:28.814175 7f29191ed700  1 mon.magna102@0(leader).log v21111 check_sub sending message to client.? 10.8.128.102:0/4257723194 with 1 entries (version 21111)
>2018-07-11 11:21:29.068198 7f29149e4700  1 mon.magna102@0(leader).log v21112 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21112)
>2018-07-11 11:21:29.068231 7f29149e4700  1 mon.magna102@0(leader).log v21112 check_sub sending message to client.? 10.8.128.102:0/4257723194 with 0 entries (version 21112)
>2018-07-11 11:21:46.652869 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:46.652892 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3979886772' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:47.444647 7f29149e4700  1 mon.magna102@0(leader).log v21113 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21113)
>2018-07-11 11:21:48.283912 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:48.283938 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1847424048' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:48.511462 7f29149e4700  1 mon.magna102@0(leader).log v21114 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21114)
>2018-07-11 11:21:49.679019 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:49.679043 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2729555385' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:49.795214 7f29149e4700  1 mon.magna102@0(leader).log v21115 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21115)
>2018-07-11 11:21:51.018285 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:51.018308 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4235331695' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:51.928893 7f29149e4700  1 mon.magna102@0(leader).log v21116 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21116)
>2018-07-11 11:21:52.301387 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:52.301410 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/476342420' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:52.995715 7f29149e4700  1 mon.magna102@0(leader).log v21117 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21117)
>2018-07-11 11:21:53.596348 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:53.596371 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1855251555' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:54.129472 7f29149e4700  1 mon.magna102@0(leader).log v21118 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21118)
>2018-07-11 11:21:56.254235 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:56.254258 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1125134676' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:56.339301 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:21:56.372728 7f29149e4700  1 mon.magna102@0(leader).log v21119 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21119)
>2018-07-11 11:21:57.439708 7f29149e4700  1 mon.magna102@0(leader).log v21120 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21120)
>2018-07-11 11:21:57.611244 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:57.611267 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1022367674' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:21:58.473239 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 9)
>2018-07-11 11:21:58.506625 7f29149e4700  1 mon.magna102@0(leader).log v21121 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21121)
>2018-07-11 11:21:59.573643 7f29149e4700  1 mon.magna102@0(leader).log v21122 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21122)
>2018-07-11 11:21:59.599036 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:21:59.599059 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3844605737' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:22:00.607117 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)
>2018-07-11 11:22:00.640566 7f29149e4700  1 mon.magna102@0(leader).log v21123 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21123)
>2018-07-11 11:22:01.606201 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:22:01.606226 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2273207050' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:22:01.707701 7f29149e4700  1 mon.magna102@0(leader).log v21124 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21124)
>2018-07-11 11:22:02.915265 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:22:02.915288 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1321201555' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:22:02.918854 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:22:03.899742 7f29149e4700  1 mon.magna102@0(leader).log v21125 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21125)
>2018-07-11 11:22:04.266492 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:22:04.266515 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3862756707' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:22:05.033403 7f29149e4700  1 mon.magna102@0(leader).log v21126 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21126)
>2018-07-11 11:22:18.329267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:22:18.476351 7f29149e4700  1 mon.magna102@0(leader).log v21127 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21127)
>2018-07-11 11:22:24.368289 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:24.493622 7f29149e4700  1 mon.magna102@0(leader).log v21128 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21128)
>2018-07-11 11:22:29.368523 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:29.494089 7f29149e4700  1 mon.magna102@0(leader).log v21129 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21129)
>2018-07-11 11:22:34.369379 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:34.503093 7f29149e4700  1 mon.magna102@0(leader).log v21130 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21130)
>2018-07-11 11:22:42.340251 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:42.513385 7f29149e4700  1 mon.magna102@0(leader).log v21131 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21131)
>2018-07-11 11:22:49.371883 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 28 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:49.472274 7f29149e4700  1 mon.magna102@0(leader).log v21132 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21132)
>2018-07-11 11:22:54.372682 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 36 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:54.489735 7f29149e4700  1 mon.magna102@0(leader).log v21133 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21133)
>2018-07-11 11:22:59.372867 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 38 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:22:59.481837 7f29149e4700  1 mon.magna102@0(leader).log v21134 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21134)
>2018-07-11 11:23:02.919034 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 11:23:06.351191 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 40 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:06.499284 7f29149e4700  1 mon.magna102@0(leader).log v21135 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21135)
>2018-07-11 11:23:08.491544 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10558. Immutable memtables: 0.
>
>2018-07-11 11:23:08.491587 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 954] Syncing log #10555
>2018-07-11 11:23:08.508106 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.491578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:23:08.508117 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 954] Flushing memtable with next log file: 10558
>
>2018-07-11 11:23:08.508131 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188508124, "job": 954, "event": "flush_started", "num_memtables": 1, "num_entries": 2122, "num_deletes": 251, "memory_usage": 4902008}
>2018-07-11 11:23:08.508134 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 954] Level-0 flush table #10559: started
>2018-07-11 11:23:08.583669 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188583653, "cf_name": "default", "job": 954, "event": "table_file_creation", "file_number": 10559, "file_size": 4492064, "table_properties": {"data_size": 4459966, "index_size": 17047, "filter_size": 14125, "raw_key_size": 17813, "raw_average_key_size": 20, "raw_value_size": 4435694, "raw_average_value_size": 5122, "num_data_blocks": 509, "num_entries": 866, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:23:08.583681 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 954] Level-0 flush table #10559: 4492064 bytes OK
>2018-07-11 11:23:08.633688 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.600187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10559 started
>2018-07-11 11:23:08.633696 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.633645) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10559: memtable #1 done
>2018-07-11 11:23:08.633698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.633660) EVENT_LOG_v1 {"time_micros": 1531308188633654, "job": 954, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:23:08.633700 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.633676) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:23:08.633707 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 954] Try to delete WAL files size 4855796, prev total WAL file size 4855796, number of live WAL files 2.
>
>2018-07-11 11:23:08.634845 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:23:08.634900 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.634893) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:23:08.634910 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 955] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:23:08.634916 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 954 Base level 0, inputs: [10559(4386KB)], [10557(14MB)]
>
>2018-07-11 11:23:08.634928 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188634919, "job": 955, "event": "compaction_started", "files_L0": [10559], "files_L1": [10557], "score": -1, "input_data_size": 19380268}
>2018-07-11 11:23:08.826062 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 955] Generated table #10560: 3438 keys, 17395178 bytes
>2018-07-11 11:23:08.826106 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188826079, "cf_name": "default", "job": 955, "event": "table_file_creation", "file_number": 10560, "file_size": 17395178, "table_properties": {"data_size": 17267655, "index_size": 71790, "filter_size": 54806, "raw_key_size": 69545, "raw_average_key_size": 20, "raw_value_size": 17168427, "raw_average_value_size": 4993, "num_data_blocks": 2091, "num_entries": 3438, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:23:08.859243 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 955] Compacted 1@0 + 1@1 files to L1 => 17395178 bytes
>2018-07-11 11:23:08.892773 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.892706) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 86.4 rd, 77.6 wr, level 1, files in(1, 1) out(1) MB in(4.3, 14.2) out(16.6), read-write-amplify(8.2) write-amplify(3.9) OK, records in: 3955, records dropped: 517
>
>2018-07-11 11:23:08.892782 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:23:08.892727) EVENT_LOG_v1 {"time_micros": 1531308188892717, "job": 955, "event": "compaction_finished", "compaction_time_micros": 224300, "output_level": 1, "num_output_files": 1, "total_output_size": 17395178, "num_input_records": 3955, "num_output_records": 3438, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:23:08.893717 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188893713, "job": 955, "event": "table_file_deletion", "file_number": 10559}
>2018-07-11 11:23:08.896574 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308188896571, "job": 955, "event": "table_file_deletion", "file_number": 10557}
>2018-07-11 11:23:12.354133 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 42 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:12.501865 7f29149e4700  1 mon.magna102@0(leader).log v21136 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21136)
>2018-07-11 11:23:19.393834 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 50 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:19.494245 7f29149e4700  1 mon.magna102@0(leader).log v21137 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21137)
>2018-07-11 11:23:24.394542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 54 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:24.512794 7f29149e4700  1 mon.magna102@0(leader).log v21138 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21138)
>2018-07-11 11:23:29.394764 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 58 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:29.496674 7f29149e4700  1 mon.magna102@0(leader).log v21139 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21139)
>2018-07-11 11:23:30.496681 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive)
>2018-07-11 11:23:30.655352 7f29149e4700  1 mon.magna102@0(leader).log v21140 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21140)
>2018-07-11 11:23:32.363024 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 58 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 11:23:32.522366 7f29149e4700  1 mon.magna102@0(leader).log v21141 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21141)
>2018-07-11 11:23:45.363954 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:23:45.363977 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1899832859' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:23:45.523424 7f29149e4700  1 mon.magna102@0(leader).log v21142 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21142)
>2018-07-11 11:23:52.372037 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:23:52.540884 7f29149e4700  1 mon.magna102@0(leader).log v21143 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21143)
>2018-07-11 11:23:59.416060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:23:59.508138 7f29149e4700  1 mon.magna102@0(leader).log v21144 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21144)
>2018-07-11 11:24:02.919229 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:25:02.919396 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:25:08.405801 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 11:25:08.556848 7f29149e4700  1 mon.magna102@0(leader).log v21145 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21145)
>2018-07-11 11:25:12.407952 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 11:25:12.565612 7f29149e4700  1 mon.magna102@0(leader).log v21146 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21146)
>2018-07-11 11:25:15.549248 7f29149e4700  1 mon.magna102@0(leader).log v21147 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 2 entries (version 21147)
>2018-07-11 11:25:19.524873 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10561. Immutable memtables: 0.
>
>2018-07-11 11:25:19.524918 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 956] Syncing log #10558
>2018-07-11 11:25:19.541430 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.524908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:25:19.541441 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 956] Flushing memtable with next log file: 10561
>
>2018-07-11 11:25:19.541455 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319541448, "job": 956, "event": "flush_started", "num_memtables": 1, "num_entries": 806, "num_deletes": 250, "memory_usage": 1266832}
>2018-07-11 11:25:19.541458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 956] Level-0 flush table #10562: started
>2018-07-11 11:25:19.583622 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319583608, "cf_name": "default", "job": 956, "event": "table_file_creation", "file_number": 10562, "file_size": 801010, "table_properties": {"data_size": 793887, "index_size": 3122, "filter_size": 3082, "raw_key_size": 7893, "raw_average_key_size": 21, "raw_value_size": 786631, "raw_average_value_size": 2149, "num_data_blocks": 97, "num_entries": 366, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:25:19.583634 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 956] Level-0 flush table #10562: 801010 bytes OK
>2018-07-11 11:25:19.633545 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.600063) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10562 started
>2018-07-11 11:25:19.633553 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.633508) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10562: memtable #1 done
>2018-07-11 11:25:19.633556 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.633523) EVENT_LOG_v1 {"time_micros": 1531308319633517, "job": 956, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:25:19.633558 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.633535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:25:19.633564 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 956] Try to delete WAL files size 1249842, prev total WAL file size 1249842, number of live WAL files 2.
>
>2018-07-11 11:25:19.633952 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:25:19.634006 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.633999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:25:19.634015 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 957] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:25:19.634021 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 956 Base level 0, inputs: [10562(782KB)], [10560(16MB)]
>
>2018-07-11 11:25:19.634034 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319634024, "job": 957, "event": "compaction_started", "files_L0": [10562], "files_L1": [10560], "score": -1, "input_data_size": 18196188}
>2018-07-11 11:25:19.835616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 957] Generated table #10563: 3355 keys, 16877483 bytes
>2018-07-11 11:25:19.835666 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319835638, "cf_name": "default", "job": 957, "event": "table_file_creation", "file_number": 10563, "file_size": 16877483, "table_properties": {"data_size": 16755243, "index_size": 68331, "filter_size": 52982, "raw_key_size": 67527, "raw_average_key_size": 20, "raw_value_size": 16659687, "raw_average_value_size": 4965, "num_data_blocks": 1996, "num_entries": 3355, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:25:19.877123 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 957] Compacted 1@0 + 1@1 files to L1 => 16877483 bytes
>2018-07-11 11:25:19.910667 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.910556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.9 rd, 69.4 wr, level 1, files in(1, 1) out(1) MB in(0.8, 16.6) out(16.1), read-write-amplify(43.8) write-amplify(21.1) OK, records in: 3804, records dropped: 449
>
>2018-07-11 11:25:19.910676 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:25:19.910577) EVENT_LOG_v1 {"time_micros": 1531308319910568, "job": 957, "event": "compaction_finished", "compaction_time_micros": 243072, "output_level": 1, "num_output_files": 1, "total_output_size": 16877483, "num_input_records": 3804, "num_output_records": 3355, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:25:19.910889 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319910884, "job": 957, "event": "table_file_deletion", "file_number": 10562}
>2018-07-11 11:25:19.914039 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308319914036, "job": 957, "event": "table_file_deletion", "file_number": 10560}
>2018-07-11 11:25:19.914091 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:25:30.991459 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:25:30.991482 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4142498056' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:25:31.578796 7f29149e4700  1 mon.magna102@0(leader).log v21148 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21148)
>2018-07-11 11:25:32.578812 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:25:32.754155 7f29149e4700  1 mon.magna102@0(leader).log v21149 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21149)
>2018-07-11 11:25:34.514504 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:25:34.514530 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/580935325' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:25:34.515791 7f29191ed700  1 mon.magna102@0(leader).log v21149 check_sub sending message to client.? 10.8.128.102:0/580935325 with 1 entries (version 21149)
>2018-07-11 11:25:35.571190 7f29149e4700  1 mon.magna102@0(leader).log v21150 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21150)
>2018-07-11 11:25:35.571223 7f29149e4700  1 mon.magna102@0(leader).log v21150 check_sub sending message to client.? 10.8.128.102:0/580935325 with 0 entries (version 21150)
>2018-07-11 11:25:39.492075 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:25:39.605077 7f29149e4700  1 mon.magna102@0(leader).log v21151 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21151)
>2018-07-11 11:25:39.605109 7f29149e4700  1 mon.magna102@0(leader).log v21151 check_sub sending message to client.? 10.8.128.102:0/580935325 with 1 entries (version 21151)
>2018-07-11 11:26:02.919570 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:26:31.576794 7f29149e4700  1 mon.magna102@0(leader).log v21152 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21152)
>2018-07-11 11:26:35.585558 7f29149e4700  1 mon.magna102@0(leader).log v21153 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21153)
>2018-07-11 11:26:46.450626 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:26:46.603418 7f29149e4700  1 mon.magna102@0(leader).log v21154 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21154)
>2018-07-11 11:26:47.520690 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:26:47.520713 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4092773135' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:26:47.678595 7f29149e4700  1 mon.magna102@0(leader).log v21155 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21155)
>2018-07-11 11:26:48.678604 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 9)
>2018-07-11 11:26:48.845616 7f29149e4700  1 mon.magna102@0(leader).log v21156 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21156)
>2018-07-11 11:26:50.172290 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:26:50.172317 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3903505640' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:26:50.173657 7f29191ed700  1 mon.magna102@0(leader).log v21156 check_sub sending message to client.? 10.8.128.102:0/3903505640 with 1 entries (version 21156)
>2018-07-11 11:26:50.304107 7f29149e4700  1 mon.magna102@0(leader).log v21157 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21157)
>2018-07-11 11:26:50.304139 7f29149e4700  1 mon.magna102@0(leader).log v21157 check_sub sending message to client.? 10.8.128.102:0/3903505640 with 0 entries (version 21157)
>2018-07-11 11:27:01.621774 7f29149e4700  1 mon.magna102@0(leader).log v21158 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21158)
>2018-07-11 11:27:02.919756 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:27:03.310529 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:27:03.310552 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/949268384' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:27:03.755467 7f29149e4700  1 mon.magna102@0(leader).log v21159 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21159)
>2018-07-11 11:27:08.537873 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 11:27:08.537900 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4278976924' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 11:27:09.614443 7f29149e4700  1 mon.magna102@0(leader).log v21160 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21160)
>2018-07-11 11:27:12.463206 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:27:12.616203 7f29149e4700  1 mon.magna102@0(leader).log v21161 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21161)
>2018-07-11 11:27:21.600158 7f29149e4700  1 mon.magna102@0(leader).log v21162 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21162)
>2018-07-11 11:27:28.196595 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:27:28.196621 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4224007919' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:27:28.197976 7f29191ed700  1 mon.magna102@0(leader).log v21162 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21162)
>2018-07-11 11:27:28.309117 7f29149e4700  1 mon.magna102@0(leader).log v21163 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21163)
>2018-07-11 11:27:28.309150 7f29149e4700  1 mon.magna102@0(leader).log v21163 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 0 entries (version 21163)
>2018-07-11 11:28:02.919929 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:28:22.494721 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 8 (REQUEST_SLOW)
>2018-07-11 11:28:22.647968 7f29149e4700  1 mon.magna102@0(leader).log v21164 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21164)
>2018-07-11 11:28:22.648015 7f29149e4700  1 mon.magna102@0(leader).log v21164 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21164)
>2018-07-11 11:28:26.496493 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 8)
>2018-07-11 11:28:26.648402 7f29149e4700  1 mon.magna102@0(leader).log v21165 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21165)
>2018-07-11 11:28:26.648446 7f29149e4700  1 mon.magna102@0(leader).log v21165 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21165)
>2018-07-11 11:28:48.506083 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 11:28:48.675077 7f29149e4700  1 mon.magna102@0(leader).log v21166 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21166)
>2018-07-11 11:28:48.675122 7f29149e4700  1 mon.magna102@0(leader).log v21166 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21166)
>2018-07-11 11:28:54.558807 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 8,9 (REQUEST_SLOW)
>2018-07-11 11:28:54.692505 7f29149e4700  1 mon.magna102@0(leader).log v21167 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21167)
>2018-07-11 11:28:54.692551 7f29149e4700  1 mon.magna102@0(leader).log v21167 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21167)
>2018-07-11 11:29:02.920096 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:29:12.516846 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 8,9)
>2018-07-11 11:29:12.668900 7f29149e4700  1 mon.magna102@0(leader).log v21168 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21168)
>2018-07-11 11:29:12.668946 7f29149e4700  1 mon.magna102@0(leader).log v21168 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 1 entries (version 21168)
>2018-07-11 11:29:13.794018 7f29149e4700  1 mon.magna102@0(leader).log v21169 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21169)
>2018-07-11 11:29:13.794052 7f29149e4700  1 mon.magna102@0(leader).log v21169 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 0 entries (version 21169)
>2018-07-11 11:29:21.653194 7f29149e4700  1 mon.magna102@0(leader).log v21170 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21170)
>2018-07-11 11:29:21.653214 7f29149e4700  1 mon.magna102@0(leader).log v21170 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 0 entries (version 21170)
>2018-07-11 11:29:31.655370 7f29149e4700  1 mon.magna102@0(leader).log v21171 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21171)
>2018-07-11 11:29:31.655390 7f29149e4700  1 mon.magna102@0(leader).log v21171 check_sub sending message to client.? 10.8.128.102:0/4224007919 with 0 entries (version 21171)
>2018-07-11 11:29:38.294130 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:29:38.294156 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1061043443' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:29:38.295437 7f29191ed700  1 mon.magna102@0(leader).log v21171 check_sub sending message to client.? 10.8.128.102:0/1061043443 with 1 entries (version 21171)
>2018-07-11 11:29:38.414331 7f29149e4700  1 mon.magna102@0(leader).log v21172 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21172)
>2018-07-11 11:29:38.414352 7f29149e4700  1 mon.magna102@0(leader).log v21172 check_sub sending message to client.? 10.8.128.102:0/1061043443 with 0 entries (version 21172)
>2018-07-11 11:29:51.673805 7f29149e4700  1 mon.magna102@0(leader).log v21173 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21173)
>2018-07-11 11:29:51.673826 7f29149e4700  1 mon.magna102@0(leader).log v21173 check_sub sending message to client.? 10.8.128.102:0/1061043443 with 0 entries (version 21173)
>2018-07-11 11:30:01.691637 7f29149e4700  1 mon.magna102@0(leader).log v21174 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21174)
>2018-07-11 11:30:01.691659 7f29149e4700  1 mon.magna102@0(leader).log v21174 check_sub sending message to client.? 10.8.128.102:0/1061043443 with 0 entries (version 21174)
>2018-07-11 11:30:02.920271 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:30:08.676006 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10564. Immutable memtables: 0.
>
>2018-07-11 11:30:08.676051 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 958] Syncing log #10561
>2018-07-11 11:30:08.692453 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.676041) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:30:08.692465 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 958] Flushing memtable with next log file: 10564
>
>2018-07-11 11:30:08.692479 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308608692471, "job": 958, "event": "flush_started", "num_memtables": 1, "num_entries": 1487, "num_deletes": 251, "memory_usage": 2780512}
>2018-07-11 11:30:08.692482 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 958] Level-0 flush table #10565: started
>2018-07-11 11:30:08.759755 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308608759740, "cf_name": "default", "job": 958, "event": "table_file_creation", "file_number": 10565, "file_size": 2660279, "table_properties": {"data_size": 2638903, "index_size": 11484, "filter_size": 8968, "raw_key_size": 12984, "raw_average_key_size": 20, "raw_value_size": 2621900, "raw_average_value_size": 4155, "num_data_blocks": 347, "num_entries": 631, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:30:08.759766 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 958] Level-0 flush table #10565: 2660279 bytes OK
>2018-07-11 11:30:08.818123 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.776305) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10565 started
>2018-07-11 11:30:08.818131 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.818086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10565: memtable #1 done
>2018-07-11 11:30:08.818133 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.818101) EVENT_LOG_v1 {"time_micros": 1531308608818094, "job": 958, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:30:08.818135 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.818113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:30:08.818142 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 958] Try to delete WAL files size 2751182, prev total WAL file size 2751182, number of live WAL files 2.
>
>2018-07-11 11:30:08.818818 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:30:08.818997 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 11:30:08.819003 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 70635.7 total, 789.3 interval
>Cumulative writes: 95K writes, 450K keys, 95K commit groups, 1.0 writes per commit group, ingest: 0.75 GB, 0.01 MB/s
>Cumulative WAL: 95K writes, 95K syncs, 1.00 writes per sync, written: 0.75 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1125 writes, 5090 keys, 1125 commit groups, 1.0 writes per commit group, ingest: 9.85 MB, 0.01 MB/s
>Interval WAL: 1125 writes, 1125 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.54 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.2        39       480    0.081       0      0
>  L1      1/1   16.10 MB   0.0      7.6     0.6      6.9       6.9      0.0       0.0  10.7     72.7     66.5       106       477    0.223   1710K   205K
> Sum      2/2   18.63 MB   0.0      7.6     0.6      6.9       7.6      0.7       0.0  11.6     53.3     53.3       145       957    0.152   1710K   205K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.9     55.9     55.8         2        11    0.142     19K   3003
>Uptime(secs): 70635.7 total, 70635.7 interval
>Flush(GB): cumulative 0.650, interval 0.009
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.56 GB write, 0.11 MB/s write, 7.55 GB read, 0.11 MB/s read, 145.2 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.09 GB read, 0.00 MB/s read, 1.6 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.54 MB   0.0      0.0     0.0      0.0       0.6      0.6       0.0   1.0      0.0     17.2        39       480    0.081       0      0
>  L1      1/1   16.10 MB   0.0      7.6     0.6      6.9       6.9      0.0       0.0  10.7     72.7     66.5       106       477    0.223   1710K   205K
> Sum      2/2   18.63 MB   0.0      7.6     0.6      6.9       7.6      0.7       0.0  11.6     53.3     53.3       145       957    0.152   1710K   205K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 70635.7 total, 70635.7 interval
>Flush(GB): cumulative 0.650, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.56 GB write, 0.11 MB/s write, 7.55 GB read, 0.11 MB/s read, 145.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 11:30:08.819023 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:30:08.819017) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:30:08.819028 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 959] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:30:08.819033 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 958 Base level 0, inputs: [10565(2597KB)], [10563(16MB)]
>
>2018-07-11 11:30:08.819048 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308608819039, "job": 959, "event": "compaction_started", "files_L0": [10565], "files_L1": [10563], "score": -1, "input_data_size": 19537762}
>2018-07-11 11:30:09.010485 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 959] Generated table #10566: 3476 keys, 17063616 bytes
>2018-07-11 11:30:09.010516 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308609010502, "cf_name": "default", "job": 959, "event": "table_file_creation", "file_number": 10566, "file_size": 17063616, "table_properties": {"data_size": 16936587, "index_size": 71852, "filter_size": 54250, "raw_key_size": 70264, "raw_average_key_size": 20, "raw_value_size": 16836775, "raw_average_value_size": 4843, "num_data_blocks": 2092, "num_entries": 3476, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:30:09.043670 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 959] Compacted 1@0 + 1@1 files to L1 => 17063616 bytes
>2018-07-11 11:30:09.085543 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:30:09.085474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 87.0 rd, 76.0 wr, level 1, files in(1, 1) out(1) MB in(2.5, 16.1) out(16.3), read-write-amplify(13.8) write-amplify(6.4) OK, records in: 3986, records dropped: 510
>
>2018-07-11 11:30:09.085552 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:30:09.085495) EVENT_LOG_v1 {"time_micros": 1531308609085485, "job": 959, "event": "compaction_finished", "compaction_time_micros": 224605, "output_level": 1, "num_output_files": 1, "total_output_size": 17063616, "num_input_records": 3986, "num_output_records": 3476, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:30:09.086087 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308609086083, "job": 959, "event": "table_file_deletion", "file_number": 10565}
>2018-07-11 11:30:09.089354 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308609089351, "job": 959, "event": "table_file_deletion", "file_number": 10563}
>2018-07-11 11:30:41.689646 7f29149e4700  1 mon.magna102@0(leader).log v21175 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21175)
>2018-07-11 11:31:02.920448 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:31:39.153507 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:31:39.153529 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/242971617' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:31:39.720176 7f29149e4700  1 mon.magna102@0(leader).log v21176 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21176)
>2018-07-11 11:31:41.637000 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:31:41.637025 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1546269245' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:31:41.638309 7f29191ed700  1 mon.magna102@0(leader).log v21176 check_sub sending message to client.? 10.8.128.102:0/1546269245 with 1 entries (version 21176)
>2018-07-11 11:31:41.862216 7f29149e4700  1 mon.magna102@0(leader).log v21177 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21177)
>2018-07-11 11:31:41.862248 7f29149e4700  1 mon.magna102@0(leader).log v21177 check_sub sending message to client.? 10.8.128.102:0/1546269245 with 0 entries (version 21177)
>2018-07-11 11:32:02.920635 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:32:05.245448 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:32:05.245471 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2767474808' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:32:05.731341 7f29149e4700  1 mon.magna102@0(leader).log v21178 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21178)
>2018-07-11 11:32:07.277148 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:32:07.277173 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2596967717' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:32:07.278432 7f29191ed700  1 mon.magna102@0(leader).log v21178 check_sub sending message to client.? 10.8.128.102:0/2596967717 with 1 entries (version 21178)
>2018-07-11 11:32:07.865032 7f29149e4700  1 mon.magna102@0(leader).log v21179 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21179)
>2018-07-11 11:32:07.865066 7f29149e4700  1 mon.magna102@0(leader).log v21179 check_sub sending message to client.? 10.8.128.102:0/2596967717 with 0 entries (version 21179)
>2018-07-11 11:33:02.920815 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:33:31.199324 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:33:31.199347 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2839510958' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:33:31.790316 7f29149e4700  1 mon.magna102@0(leader).log v21180 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21180)
>2018-07-11 11:33:32.823699 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:33:32.823728 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/113615847' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:33:32.949005 7f29149e4700  1 mon.magna102@0(leader).log v21181 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21181)
>2018-07-11 11:33:39.733163 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10567. Immutable memtables: 0.
>
>2018-07-11 11:33:39.733222 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 960] Syncing log #10564
>2018-07-11 11:33:39.749684 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.733212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:33:39.749695 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 960] Flushing memtable with next log file: 10567
>
>2018-07-11 11:33:39.749722 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308819749714, "job": 960, "event": "flush_started", "num_memtables": 1, "num_entries": 1046, "num_deletes": 250, "memory_usage": 1731744}
>2018-07-11 11:33:39.749725 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 960] Level-0 flush table #10568: started
>2018-07-11 11:33:39.800236 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308819800221, "cf_name": "default", "job": 960, "event": "table_file_creation", "file_number": 10568, "file_size": 980780, "table_properties": {"data_size": 972110, "index_size": 4063, "filter_size": 3688, "raw_key_size": 8388, "raw_average_key_size": 21, "raw_value_size": 963827, "raw_average_value_size": 2465, "num_data_blocks": 127, "num_entries": 391, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:33:39.800246 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 960] Level-0 flush table #10568: 980780 bytes OK
>2018-07-11 11:33:39.858588 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.816753) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10568 started
>2018-07-11 11:33:39.858596 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.858552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10568: memtable #1 done
>2018-07-11 11:33:39.858598 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.858566) EVENT_LOG_v1 {"time_micros": 1531308819858560, "job": 960, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:33:39.858613 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.858578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:33:39.858619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 960] Try to delete WAL files size 1710622, prev total WAL file size 1710622, number of live WAL files 2.
>
>2018-07-11 11:33:39.859114 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:33:39.859176 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:33:39.859170) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:33:39.859195 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 961] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:33:39.859201 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 960 Base level 0, inputs: [10568(957KB)], [10566(16MB)]
>
>2018-07-11 11:33:39.859215 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308819859204, "job": 961, "event": "compaction_started", "files_L0": [10568], "files_L1": [10566], "score": -1, "input_data_size": 18044396}
>2018-07-11 11:33:40.050973 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 961] Generated table #10569: 3458 keys, 17005148 bytes
>2018-07-11 11:33:40.051016 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308820050990, "cf_name": "default", "job": 961, "event": "table_file_creation", "file_number": 10569, "file_size": 17005148, "table_properties": {"data_size": 16879531, "index_size": 70760, "filter_size": 53930, "raw_key_size": 69621, "raw_average_key_size": 20, "raw_value_size": 16780821, "raw_average_value_size": 4852, "num_data_blocks": 2067, "num_entries": 3458, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:33:40.084153 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 961] Compacted 1@0 + 1@1 files to L1 => 17005148 bytes
>2018-07-11 11:33:40.126007 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:33:40.125939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.2 rd, 75.6 wr, level 1, files in(1, 1) out(1) MB in(0.9, 16.3) out(16.2), read-write-amplify(35.7) write-amplify(17.3) OK, records in: 3867, records dropped: 409
>
>2018-07-11 11:33:40.126016 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:33:40.125961) EVENT_LOG_v1 {"time_micros": 1531308820125951, "job": 961, "event": "compaction_finished", "compaction_time_micros": 224919, "output_level": 1, "num_output_files": 1, "total_output_size": 17005148, "num_input_records": 3867, "num_output_records": 3458, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:33:40.126293 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308820126289, "job": 961, "event": "table_file_deletion", "file_number": 10568}
>2018-07-11 11:33:40.129316 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531308820129300, "job": 961, "event": "table_file_deletion", "file_number": 10566}
>2018-07-11 11:33:40.129351 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:33:55.162793 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:33:55.162817 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2366918167' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:33:55.777780 7f29149e4700  1 mon.magna102@0(leader).log v21182 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21182)
>2018-07-11 11:33:56.414582 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:33:56.414604 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3558817300' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:33:56.911433 7f29149e4700  1 mon.magna102@0(leader).log v21183 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21183)
>2018-07-11 11:33:57.666562 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:33:57.666585 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3012704008' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:33:57.978319 7f29149e4700  1 mon.magna102@0(leader).log v21184 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21184)
>2018-07-11 11:34:00.167221 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:34:00.167247 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2273187733' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:34:00.168588 7f29191ed700  1 mon.magna102@0(leader).log v21184 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 1 entries (version 21184)
>2018-07-11 11:34:00.287091 7f29149e4700  1 mon.magna102@0(leader).log v21185 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21185)
>2018-07-11 11:34:00.287124 7f29149e4700  1 mon.magna102@0(leader).log v21185 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21185)
>2018-07-11 11:34:02.920994 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:35:02.921159 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:36:02.921353 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:37:02.921533 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:37:13.665713 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:37:13.665735 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/2131373757' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:37:13.862535 7f29149e4700  1 mon.magna102@0(leader).log v21186 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21186)
>2018-07-11 11:37:13.862569 7f29149e4700  1 mon.magna102@0(leader).log v21186 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21186)
>2018-07-11 11:37:29.863762 7f29149e4700  1 mon.magna102@0(leader).log v21187 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21187)
>2018-07-11 11:37:29.863784 7f29149e4700  1 mon.magna102@0(leader).log v21187 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21187)
>2018-07-11 11:37:38.006211 7f29149e4700  1 mon.magna102@0(leader).log v21188 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21188)
>2018-07-11 11:37:38.006245 7f29149e4700  1 mon.magna102@0(leader).log v21188 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21188)
>2018-07-11 11:37:44.744020 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:37:44.898753 7f29149e4700  1 mon.magna102@0(leader).log v21189 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21189)
>2018-07-11 11:37:44.898797 7f29149e4700  1 mon.magna102@0(leader).log v21189 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 1 entries (version 21189)
>2018-07-11 11:37:49.781855 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:37:49.890792 7f29149e4700  1 mon.magna102@0(leader).log v21190 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21190)
>2018-07-11 11:37:49.890838 7f29149e4700  1 mon.magna102@0(leader).log v21190 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 1 entries (version 21190)
>2018-07-11 11:37:54.109842 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:37:54.109862 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/182754402' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:37:54.242446 7f29149e4700  1 mon.magna102@0(leader).log v21191 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21191)
>2018-07-11 11:37:54.242478 7f29149e4700  1 mon.magna102@0(leader).log v21191 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21191)
>2018-07-11 11:37:54.326301 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10570. Immutable memtables: 0.
>
>2018-07-11 11:37:54.326348 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 962] Syncing log #10567
>2018-07-11 11:37:54.342806 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.326338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:37:54.342817 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 962] Flushing memtable with next log file: 10570
>
>2018-07-11 11:37:54.342844 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074342836, "job": 962, "event": "flush_started", "num_memtables": 1, "num_entries": 1225, "num_deletes": 251, "memory_usage": 2120432}
>2018-07-11 11:37:54.342847 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 962] Level-0 flush table #10571: started
>2018-07-11 11:37:54.401684 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074401670, "cf_name": "default", "job": 962, "event": "table_file_creation", "file_number": 10571, "file_size": 2028933, "table_properties": {"data_size": 2011876, "index_size": 9067, "filter_size": 7070, "raw_key_size": 11191, "raw_average_key_size": 20, "raw_value_size": 1997801, "raw_average_value_size": 3672, "num_data_blocks": 276, "num_entries": 544, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:37:54.401694 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 962] Level-0 flush table #10571: 2028933 bytes OK
>2018-07-11 11:37:54.451681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.418211) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10571 started
>2018-07-11 11:37:54.451689 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.451644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10571: memtable #1 done
>2018-07-11 11:37:54.451691 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.451659) EVENT_LOG_v1 {"time_micros": 1531309074451653, "job": 962, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:37:54.451706 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.451671) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:37:54.451713 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 962] Try to delete WAL files size 2095636, prev total WAL file size 2095636, number of live WAL files 2.
>
>2018-07-11 11:37:54.452271 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:37:54.452323 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.452316) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:37:54.452333 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 963] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:37:54.452339 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 962 Base level 0, inputs: [10571(1981KB)], [10569(16MB)]
>
>2018-07-11 11:37:54.452352 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074452342, "job": 963, "event": "compaction_started", "files_L0": [10571], "files_L1": [10569], "score": -1, "input_data_size": 19034081}
>2018-07-11 11:37:54.644067 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 963] Generated table #10572: 3492 keys, 16578335 bytes
>2018-07-11 11:37:54.644111 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074644084, "cf_name": "default", "job": 963, "event": "table_file_creation", "file_number": 10572, "file_size": 16578335, "table_properties": {"data_size": 16452168, "index_size": 71890, "filter_size": 53350, "raw_key_size": 70565, "raw_average_key_size": 20, "raw_value_size": 16352105, "raw_average_value_size": 4682, "num_data_blocks": 2093, "num_entries": 3492, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:37:54.677277 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 963] Compacted 1@0 + 1@1 files to L1 => 16578335 bytes
>2018-07-11 11:37:54.710785 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.710719) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 84.6 rd, 73.7 wr, level 1, files in(1, 1) out(1) MB in(1.9, 16.2) out(15.8), read-write-amplify(17.6) write-amplify(8.2) OK, records in: 4002, records dropped: 510
>
>2018-07-11 11:37:54.710794 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:37:54.710740) EVENT_LOG_v1 {"time_micros": 1531309074710731, "job": 963, "event": "compaction_finished", "compaction_time_micros": 224908, "output_level": 1, "num_output_files": 1, "total_output_size": 16578335, "num_input_records": 4002, "num_output_records": 3492, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:37:54.711235 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074711229, "job": 963, "event": "table_file_deletion", "file_number": 10571}
>2018-07-11 11:37:54.714608 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309074714606, "job": 963, "event": "table_file_deletion", "file_number": 10569}
>2018-07-11 11:37:55.326085 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:37:55.503229 7f29149e4700  1 mon.magna102@0(leader).log v21192 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21192)
>2018-07-11 11:37:55.503276 7f29149e4700  1 mon.magna102@0(leader).log v21192 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 1 entries (version 21192)
>2018-07-11 11:37:56.959822 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:37:56.959843 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3961681829' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:37:57.903525 7f29149e4700  1 mon.magna102@0(leader).log v21193 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21193)
>2018-07-11 11:37:57.903561 7f29149e4700  1 mon.magna102@0(leader).log v21193 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 0 entries (version 21193)
>2018-07-11 11:38:00.751281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:38:00.904081 7f29149e4700  1 mon.magna102@0(leader).log v21194 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21194)
>2018-07-11 11:38:00.904126 7f29149e4700  1 mon.magna102@0(leader).log v21194 check_sub sending message to client.? 10.8.128.102:0/2273187733 with 1 entries (version 21194)
>2018-07-11 11:38:02.921716 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:38:03.734949 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 11:38:03.734975 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/831730969' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 11:38:03.736285 7f29191ed700  1 mon.magna102@0(leader).log v21194 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21194)
>2018-07-11 11:38:03.887717 7f29149e4700  1 mon.magna102@0(leader).log v21195 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21195)
>2018-07-11 11:38:03.887738 7f29149e4700  1 mon.magna102@0(leader).log v21195 check_sub sending message to client.? 10.8.128.102:0/831730969 with 0 entries (version 21195)
>2018-07-11 11:38:04.946299 7f29149e4700  1 mon.magna102@0(leader).log v21196 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21196)
>2018-07-11 11:38:04.946333 7f29149e4700  1 mon.magna102@0(leader).log v21196 check_sub sending message to client.? 10.8.128.102:0/831730969 with 0 entries (version 21196)
>2018-07-11 11:38:09.782659 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 10 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:38:09.905429 7f29149e4700  1 mon.magna102@0(leader).log v21197 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21197)
>2018-07-11 11:38:09.905475 7f29149e4700  1 mon.magna102@0(leader).log v21197 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21197)
>2018-07-11 11:38:14.789072 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 18 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:38:14.906092 7f29149e4700  1 mon.magna102@0(leader).log v21198 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21198)
>2018-07-11 11:38:14.906137 7f29149e4700  1 mon.magna102@0(leader).log v21198 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21198)
>2018-07-11 11:38:19.789298 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:38:19.906751 7f29149e4700  1 mon.magna102@0(leader).log v21199 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21199)
>2018-07-11 11:38:19.906797 7f29149e4700  1 mon.magna102@0(leader).log v21199 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21199)
>2018-07-11 11:38:24.806948 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 22 slow requests are blocked > 32 sec. Implicated osds 2,5,10 (REQUEST_SLOW)
>2018-07-11 11:38:24.941942 7f29149e4700  1 mon.magna102@0(leader).log v21200 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21200)
>2018-07-11 11:38:24.941989 7f29149e4700  1 mon.magna102@0(leader).log v21200 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21200)
>2018-07-11 11:38:54.775286 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:38:54.927709 7f29149e4700  1 mon.magna102@0(leader).log v21201 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21201)
>2018-07-11 11:38:54.927754 7f29149e4700  1 mon.magna102@0(leader).log v21201 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21201)
>2018-07-11 11:38:59.810823 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:38:59.919815 7f29149e4700  1 mon.magna102@0(leader).log v21202 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21202)
>2018-07-11 11:38:59.919861 7f29149e4700  1 mon.magna102@0(leader).log v21202 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21202)
>2018-07-11 11:39:02.921897 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:39:04.779516 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:39:04.928777 7f29149e4700  1 mon.magna102@0(leader).log v21203 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21203)
>2018-07-11 11:39:04.928823 7f29149e4700  1 mon.magna102@0(leader).log v21203 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21203)
>2018-07-11 11:39:22.787049 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:39:22.938512 7f29149e4700  1 mon.magna102@0(leader).log v21204 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21204)
>2018-07-11 11:39:22.938557 7f29149e4700  1 mon.magna102@0(leader).log v21204 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21204)
>2018-07-11 11:39:29.822004 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:39:29.931967 7f29149e4700  1 mon.magna102@0(leader).log v21205 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21205)
>2018-07-11 11:39:29.932013 7f29149e4700  1 mon.magna102@0(leader).log v21205 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21205)
>2018-07-11 11:40:02.922073 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:40:38.820316 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:40:38.972229 7f29149e4700  1 mon.magna102@0(leader).log v21206 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21206)
>2018-07-11 11:40:38.972274 7f29149e4700  1 mon.magna102@0(leader).log v21206 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21206)
>2018-07-11 11:40:44.823101 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:40:44.981144 7f29149e4700  1 mon.magna102@0(leader).log v21207 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21207)
>2018-07-11 11:40:44.981204 7f29149e4700  1 mon.magna102@0(leader).log v21207 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21207)
>2018-07-11 11:41:02.922256 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:41:04.831758 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:41:04.982667 7f29149e4700  1 mon.magna102@0(leader).log v21208 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21208)
>2018-07-11 11:41:04.982713 7f29149e4700  1 mon.magna102@0(leader).log v21208 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21208)
>2018-07-11 11:41:09.865872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:41:09.974793 7f29149e4700  1 mon.magna102@0(leader).log v21209 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21209)
>2018-07-11 11:41:09.974839 7f29149e4700  1 mon.magna102@0(leader).log v21209 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21209)
>2018-07-11 11:41:59.963237 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10573. Immutable memtables: 0.
>
>2018-07-11 11:41:59.963282 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 964] Syncing log #10570
>2018-07-11 11:41:59.979750 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:41:59.963272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:41:59.979761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 964] Flushing memtable with next log file: 10573
>
>2018-07-11 11:41:59.979787 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309319979779, "job": 964, "event": "flush_started", "num_memtables": 1, "num_entries": 1260, "num_deletes": 250, "memory_usage": 2245224}
>2018-07-11 11:41:59.979790 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 964] Level-0 flush table #10574: started
>2018-07-11 11:42:00.030306 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309320030292, "cf_name": "default", "job": 964, "event": "table_file_creation", "file_number": 10574, "file_size": 1335243, "table_properties": {"data_size": 1324158, "index_size": 5400, "filter_size": 4765, "raw_key_size": 9419, "raw_average_key_size": 21, "raw_value_size": 1314194, "raw_average_value_size": 2973, "num_data_blocks": 168, "num_entries": 442, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:42:00.030316 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 964] Level-0 flush table #10574: 1335243 bytes OK
>2018-07-11 11:42:00.080314 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.046807) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10574 started
>2018-07-11 11:42:00.080322 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.080278) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10574: memtable #1 done
>2018-07-11 11:42:00.080324 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.080292) EVENT_LOG_v1 {"time_micros": 1531309320080286, "job": 964, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:42:00.080340 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.080304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:42:00.080346 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 964] Try to delete WAL files size 2220252, prev total WAL file size 2220252, number of live WAL files 2.
>
>2018-07-11 11:42:00.080931 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:42:00.081124 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 11:42:00.081130 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 71346.9 total, 711.3 interval
>Cumulative writes: 96K writes, 454K keys, 96K commit groups, 1.0 writes per commit group, ingest: 0.76 GB, 0.01 MB/s
>Cumulative WAL: 96K writes, 96K syncs, 1.00 writes per sync, written: 0.76 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 786 writes, 3531 keys, 786 commit groups, 1.0 writes per commit group, ingest: 5.75 MB, 0.01 MB/s
>Interval WAL: 786 writes, 786 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.27 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       483    0.081       0      0
>  L1      1/1   15.81 MB   0.0      7.6     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       107       480    0.223   1722K   206K
> Sum      2/2   17.08 MB   0.0      7.6     0.7      7.0       7.6      0.7       0.0  11.6     53.3     53.3       146       963    0.152   1722K   206K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0  12.7     61.1     59.3         1         6    0.147     11K   1429
>Uptime(secs): 71346.9 total, 71346.9 interval
>Flush(GB): cumulative 0.654, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.61 GB write, 0.11 MB/s write, 7.60 GB read, 0.11 MB/s read, 146.1 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.27 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       483    0.081       0      0
>  L1      1/1   15.81 MB   0.0      7.6     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       107       480    0.223   1722K   206K
> Sum      2/2   17.08 MB   0.0      7.6     0.7      7.0       7.6      0.7       0.0  11.6     53.3     53.3       146       963    0.152   1722K   206K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 71346.9 total, 71346.9 interval
>Flush(GB): cumulative 0.654, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.61 GB write, 0.11 MB/s write, 7.60 GB read, 0.11 MB/s read, 146.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 11:42:00.081152 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.081146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:42:00.081157 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 965] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:42:00.081162 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 964 Base level 0, inputs: [10574(1303KB)], [10572(15MB)]
>
>2018-07-11 11:42:00.081174 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309320081166, "job": 965, "event": "compaction_started", "files_L0": [10574], "files_L1": [10572], "score": -1, "input_data_size": 17913578}
>2018-07-11 11:42:00.281040 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 965] Generated table #10575: 3543 keys, 17000706 bytes
>2018-07-11 11:42:00.281089 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309320281062, "cf_name": "default", "job": 965, "event": "table_file_creation", "file_number": 10575, "file_size": 17000706, "table_properties": {"data_size": 16872460, "index_size": 72806, "filter_size": 54513, "raw_key_size": 71349, "raw_average_key_size": 20, "raw_value_size": 16771105, "raw_average_value_size": 4733, "num_data_blocks": 2127, "num_entries": 3543, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:42:00.314227 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 965] Compacted 1@0 + 1@1 files to L1 => 17000706 bytes
>2018-07-11 11:42:00.347730 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.347662) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.9 rd, 73.0 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.8) out(16.2), read-write-amplify(26.1) write-amplify(12.7) OK, records in: 3934, records dropped: 391
>
>2018-07-11 11:42:00.347738 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:42:00.347684) EVENT_LOG_v1 {"time_micros": 1531309320347675, "job": 965, "event": "compaction_finished", "compaction_time_micros": 233024, "output_level": 1, "num_output_files": 1, "total_output_size": 17000706, "num_input_records": 3934, "num_output_records": 3543, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:42:00.348075 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309320348071, "job": 965, "event": "table_file_deletion", "file_number": 10574}
>2018-07-11 11:42:00.350960 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309320350957, "job": 965, "event": "table_file_deletion", "file_number": 10572}
>2018-07-11 11:42:00.351074 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:42:02.922439 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:42:18.864200 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:42:19.024677 7f29149e4700  1 mon.magna102@0(leader).log v21210 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21210)
>2018-07-11 11:42:19.024723 7f29149e4700  1 mon.magna102@0(leader).log v21210 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21210)
>2018-07-11 11:42:28.868535 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:42:29.042318 7f29149e4700  1 mon.magna102@0(leader).log v21211 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21211)
>2018-07-11 11:42:29.042364 7f29149e4700  1 mon.magna102@0(leader).log v21211 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21211)
>2018-07-11 11:42:44.875502 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:42:45.019911 7f29149e4700  1 mon.magna102@0(leader).log v21212 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21212)
>2018-07-11 11:42:45.019956 7f29149e4700  1 mon.magna102@0(leader).log v21212 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21212)
>2018-07-11 11:42:49.931443 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:42:50.020362 7f29149e4700  1 mon.magna102@0(leader).log v21213 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21213)
>2018-07-11 11:42:50.020407 7f29149e4700  1 mon.magna102@0(leader).log v21213 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21213)
>2018-07-11 11:43:02.922614 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:44:02.909256 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:44:02.951585 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:44:03.085278 7f29149e4700  1 mon.magna102@0(leader).log v21214 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21214)
>2018-07-11 11:44:03.085324 7f29149e4700  1 mon.magna102@0(leader).log v21214 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21214)
>2018-07-11 11:44:08.912001 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:44:09.085875 7f29149e4700  1 mon.magna102@0(leader).log v21215 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21215)
>2018-07-11 11:44:09.085921 7f29149e4700  1 mon.magna102@0(leader).log v21215 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21215)
>2018-07-11 11:44:22.917875 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:44:23.070332 7f29149e4700  1 mon.magna102@0(leader).log v21216 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21216)
>2018-07-11 11:44:23.070377 7f29149e4700  1 mon.magna102@0(leader).log v21216 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21216)
>2018-07-11 11:44:28.920487 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:44:29.070756 7f29149e4700  1 mon.magna102@0(leader).log v21217 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21217)
>2018-07-11 11:44:29.070801 7f29149e4700  1 mon.magna102@0(leader).log v21217 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21217)
>2018-07-11 11:45:02.966163 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:45:19.076143 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10576. Immutable memtables: 0.
>
>2018-07-11 11:45:19.076193 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 966] Syncing log #10573
>2018-07-11 11:45:19.092638 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.076177) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:45:19.092649 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 966] Flushing memtable with next log file: 10576
>
>2018-07-11 11:45:19.092675 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519092668, "job": 966, "event": "flush_started", "num_memtables": 1, "num_entries": 1020, "num_deletes": 251, "memory_usage": 1701320}
>2018-07-11 11:45:19.092678 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 966] Level-0 flush table #10577: started
>2018-07-11 11:45:19.151569 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519151555, "cf_name": "default", "job": 966, "event": "table_file_creation", "file_number": 10577, "file_size": 1614077, "table_properties": {"data_size": 1600261, "index_size": 7157, "filter_size": 5739, "raw_key_size": 9935, "raw_average_key_size": 20, "raw_value_size": 1588422, "raw_average_value_size": 3281, "num_data_blocks": 218, "num_entries": 484, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:45:19.151580 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 966] Level-0 flush table #10577: 1614077 bytes OK
>2018-07-11 11:45:19.201602 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.168092) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10577 started
>2018-07-11 11:45:19.201610 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.201566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10577: memtable #1 done
>2018-07-11 11:45:19.201613 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.201580) EVENT_LOG_v1 {"time_micros": 1531309519201574, "job": 966, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:45:19.201614 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.201592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:45:19.201634 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 966] Try to delete WAL files size 1680243, prev total WAL file size 1680243, number of live WAL files 2.
>
>2018-07-11 11:45:19.202109 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:45:19.202171 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.202164) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:45:19.202177 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 967] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:45:19.202194 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 966 Base level 0, inputs: [10577(1576KB)], [10575(16MB)]
>
>2018-07-11 11:45:19.202210 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519202198, "job": 967, "event": "compaction_started", "files_L0": [10577], "files_L1": [10575], "score": -1, "input_data_size": 18614783}
>2018-07-11 11:45:19.393970 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 967] Generated table #10578: 3517 keys, 16584924 bytes
>2018-07-11 11:45:19.394019 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519393990, "cf_name": "default", "job": 967, "event": "table_file_creation", "file_number": 10578, "file_size": 16584924, "table_properties": {"data_size": 16458666, "index_size": 71912, "filter_size": 53419, "raw_key_size": 71037, "raw_average_key_size": 20, "raw_value_size": 16358240, "raw_average_value_size": 4651, "num_data_blocks": 2093, "num_entries": 3517, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:45:19.427164 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 967] Compacted 1@0 + 1@1 files to L1 => 16584924 bytes
>2018-07-11 11:45:19.460679 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.460613) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.8 rd, 73.7 wr, level 1, files in(1, 1) out(1) MB in(1.5, 16.2) out(15.8), read-write-amplify(21.8) write-amplify(10.3) OK, records in: 4027, records dropped: 510
>
>2018-07-11 11:45:19.460688 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:45:19.460635) EVENT_LOG_v1 {"time_micros": 1531309519460625, "job": 967, "event": "compaction_finished", "compaction_time_micros": 224937, "output_level": 1, "num_output_files": 1, "total_output_size": 16584924, "num_input_records": 4027, "num_output_records": 3517, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:45:19.461105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519461101, "job": 967, "event": "table_file_deletion", "file_number": 10577}
>2018-07-11 11:45:19.464285 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309519464282, "job": 967, "event": "table_file_deletion", "file_number": 10575}
>2018-07-11 11:45:38.950998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:45:39.104395 7f29149e4700  1 mon.magna102@0(leader).log v21218 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21218)
>2018-07-11 11:45:39.104441 7f29149e4700  1 mon.magna102@0(leader).log v21218 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21218)
>2018-07-11 11:45:48.955254 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:45:49.123152 7f29149e4700  1 mon.magna102@0(leader).log v21219 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21219)
>2018-07-11 11:45:49.123210 7f29149e4700  1 mon.magna102@0(leader).log v21219 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21219)
>2018-07-11 11:46:02.998941 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:46:08.963884 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:46:09.124822 7f29149e4700  1 mon.magna102@0(leader).log v21220 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21220)
>2018-07-11 11:46:09.124869 7f29149e4700  1 mon.magna102@0(leader).log v21220 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21220)
>2018-07-11 11:47:03.021264 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:47:22.995960 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:47:23.147996 7f29149e4700  1 mon.magna102@0(leader).log v21221 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21221)
>2018-07-11 11:47:23.148043 7f29149e4700  1 mon.magna102@0(leader).log v21221 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21221)
>2018-07-11 11:47:28.998566 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:47:29.156927 7f29149e4700  1 mon.magna102@0(leader).log v21222 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21222)
>2018-07-11 11:47:29.156973 7f29149e4700  1 mon.magna102@0(leader).log v21222 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21222)
>2018-07-11 11:47:45.005623 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:47:45.166532 7f29149e4700  1 mon.magna102@0(leader).log v21223 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21223)
>2018-07-11 11:47:45.166577 7f29149e4700  1 mon.magna102@0(leader).log v21223 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21223)
>2018-07-11 11:47:50.041386 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:47:50.158643 7f29149e4700  1 mon.magna102@0(leader).log v21224 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21224)
>2018-07-11 11:47:50.158689 7f29149e4700  1 mon.magna102@0(leader).log v21224 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21224)
>2018-07-11 11:48:03.051224 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:49:03.039600 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:49:03.073315 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:49:03.190417 7f29149e4700  1 mon.magna102@0(leader).log v21225 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21225)
>2018-07-11 11:49:03.190463 7f29149e4700  1 mon.magna102@0(leader).log v21225 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21225)
>2018-07-11 11:49:09.042449 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:49:09.199344 7f29149e4700  1 mon.magna102@0(leader).log v21226 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21226)
>2018-07-11 11:49:09.199389 7f29149e4700  1 mon.magna102@0(leader).log v21226 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21226)
>2018-07-11 11:49:29.051034 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:49:29.227104 7f29149e4700  1 mon.magna102@0(leader).log v21227 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21227)
>2018-07-11 11:49:29.227151 7f29149e4700  1 mon.magna102@0(leader).log v21227 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21227)
>2018-07-11 11:50:03.104628 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:50:20.173115 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10579. Immutable memtables: 0.
>
>2018-07-11 11:50:20.173158 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 968] Syncing log #10576
>2018-07-11 11:50:20.190767 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.173149) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:50:20.190778 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 968] Flushing memtable with next log file: 10579
>
>2018-07-11 11:50:20.190805 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820190796, "job": 968, "event": "flush_started", "num_memtables": 1, "num_entries": 1385, "num_deletes": 250, "memory_usage": 2508984}
>2018-07-11 11:50:20.190808 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 968] Level-0 flush table #10580: started
>2018-07-11 11:50:20.241329 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820241315, "cf_name": "default", "job": 968, "event": "table_file_creation", "file_number": 10580, "file_size": 1395469, "table_properties": {"data_size": 1383894, "index_size": 5697, "filter_size": 4958, "raw_key_size": 9471, "raw_average_key_size": 21, "raw_value_size": 1373676, "raw_average_value_size": 3086, "num_data_blocks": 178, "num_entries": 445, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:50:20.241339 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 968] Level-0 flush table #10580: 1395469 bytes OK
>2018-07-11 11:50:20.299670 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.257840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10580 started
>2018-07-11 11:50:20.299678 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.299634) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10580: memtable #1 done
>2018-07-11 11:50:20.299680 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.299649) EVENT_LOG_v1 {"time_micros": 1531309820299643, "job": 968, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:50:20.299682 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.299660) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:50:20.299701 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 968] Try to delete WAL files size 2482213, prev total WAL file size 2482213, number of live WAL files 2.
>
>2018-07-11 11:50:20.300357 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:50:20.300408 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.300400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:50:20.300417 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 969] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:50:20.300423 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 968 Base level 0, inputs: [10580(1362KB)], [10578(15MB)]
>
>2018-07-11 11:50:20.300438 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820300428, "job": 969, "event": "compaction_started", "files_L0": [10580], "files_L1": [10578], "score": -1, "input_data_size": 17980393}
>2018-07-11 11:50:20.508721 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 969] Generated table #10581: 3598 keys, 17248324 bytes
>2018-07-11 11:50:20.508772 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820508743, "cf_name": "default", "job": 969, "event": "table_file_creation", "file_number": 10581, "file_size": 17248324, "table_properties": {"data_size": 17117919, "index_size": 74106, "filter_size": 55372, "raw_key_size": 72467, "raw_average_key_size": 20, "raw_value_size": 17014870, "raw_average_value_size": 4728, "num_data_blocks": 2165, "num_entries": 3598, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:50:20.550236 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 969] Compacted 1@0 + 1@1 files to L1 => 17248324 bytes
>2018-07-11 11:50:20.583766 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.583696) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.8) out(16.4), read-write-amplify(25.2) write-amplify(12.4) OK, records in: 3962, records dropped: 364
>
>2018-07-11 11:50:20.583775 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:50:20.583718) EVENT_LOG_v1 {"time_micros": 1531309820583708, "job": 969, "event": "compaction_finished", "compaction_time_micros": 249781, "output_level": 1, "num_output_files": 1, "total_output_size": 17248324, "num_input_records": 3962, "num_output_records": 3598, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:50:20.584137 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820584133, "job": 969, "event": "table_file_deletion", "file_number": 10580}
>2018-07-11 11:50:20.587051 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309820587048, "job": 969, "event": "table_file_deletion", "file_number": 10578}
>2018-07-11 11:50:20.587169 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:50:43.083705 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:50:43.236297 7f29149e4700  1 mon.magna102@0(leader).log v21228 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21228)
>2018-07-11 11:50:43.236343 7f29149e4700  1 mon.magna102@0(leader).log v21228 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21228)
>2018-07-11 11:50:49.086522 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:50:49.236827 7f29149e4700  1 mon.magna102@0(leader).log v21229 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21229)
>2018-07-11 11:50:49.236873 7f29149e4700  1 mon.magna102@0(leader).log v21229 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21229)
>2018-07-11 11:51:03.129314 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:51:05.093485 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:51:05.246504 7f29149e4700  1 mon.magna102@0(leader).log v21230 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21230)
>2018-07-11 11:51:05.246550 7f29149e4700  1 mon.magna102@0(leader).log v21230 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21230)
>2018-07-11 11:51:10.141223 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:51:10.230295 7f29149e4700  1 mon.magna102@0(leader).log v21231 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21231)
>2018-07-11 11:51:10.230341 7f29149e4700  1 mon.magna102@0(leader).log v21231 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21231)
>2018-07-11 11:52:03.160123 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:52:23.127497 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:52:23.296410 7f29149e4700  1 mon.magna102@0(leader).log v21232 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21232)
>2018-07-11 11:52:23.296456 7f29149e4700  1 mon.magna102@0(leader).log v21232 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21232)
>2018-07-11 11:52:35.132995 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:52:35.280726 7f29149e4700  1 mon.magna102@0(leader).log v21233 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21233)
>2018-07-11 11:52:35.280772 7f29149e4700  1 mon.magna102@0(leader).log v21233 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21233)
>2018-07-11 11:52:49.139002 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:52:49.290171 7f29149e4700  1 mon.magna102@0(leader).log v21234 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21234)
>2018-07-11 11:52:49.290229 7f29149e4700  1 mon.magna102@0(leader).log v21234 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21234)
>2018-07-11 11:53:03.182657 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:53:03.283248 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10582. Immutable memtables: 0.
>
>2018-07-11 11:53:03.283294 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 970] Syncing log #10579
>2018-07-11 11:53:03.299783 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.283284) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:53:03.299794 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 970] Flushing memtable with next log file: 10582
>
>2018-07-11 11:53:03.299821 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983299813, "job": 970, "event": "flush_started", "num_memtables": 1, "num_entries": 886, "num_deletes": 251, "memory_usage": 1409288}
>2018-07-11 11:53:03.299824 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 970] Level-0 flush table #10583: started
>2018-07-11 11:53:03.350344 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983350329, "cf_name": "default", "job": 970, "event": "table_file_creation", "file_number": 10583, "file_size": 1337387, "table_properties": {"data_size": 1325702, "index_size": 5906, "filter_size": 4859, "raw_key_size": 9118, "raw_average_key_size": 20, "raw_value_size": 1315316, "raw_average_value_size": 2955, "num_data_blocks": 180, "num_entries": 445, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 11:53:03.350355 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 970] Level-0 flush table #10583: 1337387 bytes OK
>2018-07-11 11:53:03.408677 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.366854) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10583 started
>2018-07-11 11:53:03.408685 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.408640) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10583: memtable #1 done
>2018-07-11 11:53:03.408687 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.408655) EVENT_LOG_v1 {"time_micros": 1531309983408649, "job": 970, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:53:03.408702 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.408667) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:53:03.408709 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 970] Try to delete WAL files size 1390563, prev total WAL file size 1390563, number of live WAL files 2.
>
>2018-07-11 11:53:03.409132 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:53:03.409336 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 11:53:03.409343 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 72010.3 total, 663.3 interval
>Cumulative writes: 96K writes, 457K keys, 96K commit groups, 1.0 writes per commit group, ingest: 0.76 GB, 0.01 MB/s
>Cumulative WAL: 96K writes, 96K syncs, 1.00 writes per sync, written: 0.76 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 720 writes, 3291 keys, 720 commit groups, 1.0 writes per commit group, ingest: 5.30 MB, 0.01 MB/s
>Interval WAL: 720 writes, 720 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.28 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       486    0.081       0      0
>  L1      1/1   16.45 MB   0.0      7.7     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       108       483    0.223   1734K   208K
> Sum      2/2   17.72 MB   0.0      7.7     0.7      7.0       7.7      0.7       0.0  11.6     53.3     53.4       147       969    0.152   1734K   208K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0  12.7     56.7     57.4         1         6    0.153     11K   1265
>Uptime(secs): 72010.3 total, 72010.3 interval
>Flush(GB): cumulative 0.658, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.66 GB write, 0.11 MB/s write, 7.66 GB read, 0.11 MB/s read, 147.0 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.28 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       486    0.081       0      0
>  L1      1/1   16.45 MB   0.0      7.7     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       108       483    0.223   1734K   208K
> Sum      2/2   17.72 MB   0.0      7.7     0.7      7.0       7.7      0.7       0.0  11.6     53.3     53.4       147       969    0.152   1734K   208K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 72010.3 total, 72010.3 interval
>Flush(GB): cumulative 0.658, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.66 GB write, 0.11 MB/s write, 7.66 GB read, 0.11 MB/s read, 147.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 11:53:03.409365 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.409359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 11:53:03.409370 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 971] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:53:03.409375 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 970 Base level 0, inputs: [10583(1306KB)], [10581(16MB)]
>
>2018-07-11 11:53:03.409388 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983409379, "job": 971, "event": "compaction_started", "files_L0": [10583], "files_L1": [10581], "score": -1, "input_data_size": 18585711}
>2018-07-11 11:53:03.601038 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 971] Generated table #10584: 3533 keys, 16653590 bytes
>2018-07-11 11:53:03.601083 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983601055, "cf_name": "default", "job": 971, "event": "table_file_creation", "file_number": 10584, "file_size": 16653590, "table_properties": {"data_size": 16527155, "index_size": 71922, "filter_size": 53586, "raw_key_size": 71338, "raw_average_key_size": 20, "raw_value_size": 16426493, "raw_average_value_size": 4649, "num_data_blocks": 2093, "num_entries": 3533, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:53:03.634236 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 971] Compacted 1@0 + 1@1 files to L1 => 16653590 bytes
>2018-07-11 11:53:03.667765 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.667695) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.7 rd, 74.1 wr, level 1, files in(1, 1) out(1) MB in(1.3, 16.4) out(15.9), read-write-amplify(26.3) write-amplify(12.5) OK, records in: 4043, records dropped: 510
>
>2018-07-11 11:53:03.667774 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:53:03.667717) EVENT_LOG_v1 {"time_micros": 1531309983667707, "job": 971, "event": "compaction_finished", "compaction_time_micros": 224831, "output_level": 1, "num_output_files": 1, "total_output_size": 16653590, "num_input_records": 4043, "num_output_records": 3533, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:53:03.668145 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983668140, "job": 971, "event": "table_file_deletion", "file_number": 10583}
>2018-07-11 11:53:03.671334 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531309983671331, "job": 971, "event": "table_file_deletion", "file_number": 10581}
>2018-07-11 11:54:03.206820 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:54:09.174227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:54:09.349251 7f29149e4700  1 mon.magna102@0(leader).log v21235 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21235)
>2018-07-11 11:54:09.349297 7f29149e4700  1 mon.magna102@0(leader).log v21235 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21235)
>2018-07-11 11:54:15.176948 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:54:15.349827 7f29149e4700  1 mon.magna102@0(leader).log v21236 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21236)
>2018-07-11 11:54:15.349872 7f29149e4700  1 mon.magna102@0(leader).log v21236 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21236)
>2018-07-11 11:54:29.182902 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:54:29.327291 7f29149e4700  1 mon.magna102@0(leader).log v21237 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21237)
>2018-07-11 11:54:29.327337 7f29149e4700  1 mon.magna102@0(leader).log v21237 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21237)
>2018-07-11 11:55:03.229489 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:55:43.215035 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:55:43.367130 7f29149e4700  1 mon.magna102@0(leader).log v21238 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21238)
>2018-07-11 11:55:43.367177 7f29149e4700  1 mon.magna102@0(leader).log v21238 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21238)
>2018-07-11 11:55:55.220531 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:55:55.393118 7f29149e4700  1 mon.magna102@0(leader).log v21239 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21239)
>2018-07-11 11:55:55.393163 7f29149e4700  1 mon.magna102@0(leader).log v21239 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21239)
>2018-07-11 11:56:03.260271 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:56:07.225587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:56:07.402565 7f29149e4700  1 mon.magna102@0(leader).log v21240 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21240)
>2018-07-11 11:56:07.402611 7f29149e4700  1 mon.magna102@0(leader).log v21240 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21240)
>2018-07-11 11:56:15.229356 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:56:15.378169 7f29149e4700  1 mon.magna102@0(leader).log v21241 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21241)
>2018-07-11 11:56:15.378230 7f29149e4700  1 mon.magna102@0(leader).log v21241 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21241)
>2018-07-11 11:57:03.282691 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:57:27.260419 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:57:27.409694 7f29149e4700  1 mon.magna102@0(leader).log v21242 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21242)
>2018-07-11 11:57:27.409740 7f29149e4700  1 mon.magna102@0(leader).log v21242 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21242)
>2018-07-11 11:57:35.264253 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:57:35.418739 7f29149e4700  1 mon.magna102@0(leader).log v21243 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21243)
>2018-07-11 11:57:35.418784 7f29149e4700  1 mon.magna102@0(leader).log v21243 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21243)
>2018-07-11 11:57:53.271609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 11:57:53.446363 7f29149e4700  1 mon.magna102@0(leader).log v21244 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21244)
>2018-07-11 11:57:53.446408 7f29149e4700  1 mon.magna102@0(leader).log v21244 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21244)
>2018-07-11 11:58:00.313230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:58:00.421946 7f29149e4700  1 mon.magna102@0(leader).log v21245 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21245)
>2018-07-11 11:58:00.421992 7f29149e4700  1 mon.magna102@0(leader).log v21245 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21245)
>2018-07-11 11:58:03.330430 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 11:58:40.392063 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10585. Immutable memtables: 0.
>
>2018-07-11 11:58:40.392105 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 972] Syncing log #10582
>2018-07-11 11:58:40.408544 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.392096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 11:58:40.408556 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 972] Flushing memtable with next log file: 10585
>
>2018-07-11 11:58:40.408582 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320408574, "job": 972, "event": "flush_started", "num_memtables": 1, "num_entries": 1519, "num_deletes": 250, "memory_usage": 2815000}
>2018-07-11 11:58:40.408585 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 972] Level-0 flush table #10586: started
>2018-07-11 11:58:40.459085 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320459070, "cf_name": "default", "job": 972, "event": "table_file_creation", "file_number": 10586, "file_size": 1561979, "table_properties": {"data_size": 1549255, "index_size": 6339, "filter_size": 5465, "raw_key_size": 9892, "raw_average_key_size": 21, "raw_value_size": 1538278, "raw_average_value_size": 3301, "num_data_blocks": 198, "num_entries": 466, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 11:58:40.459095 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 972] Level-0 flush table #10586: 1561979 bytes OK
>2018-07-11 11:58:40.509092 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.475608) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10586 started
>2018-07-11 11:58:40.509100 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.509056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10586: memtable #1 done
>2018-07-11 11:58:40.509103 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.509071) EVENT_LOG_v1 {"time_micros": 1531310320509064, "job": 972, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 11:58:40.509105 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.509082) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 11:58:40.509124 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 972] Try to delete WAL files size 2785831, prev total WAL file size 2785831, number of live WAL files 2.
>
>2018-07-11 11:58:40.509826 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:58:40.509877 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.509870) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 11:58:40.509886 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 973] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 11:58:40.509892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 972 Base level 0, inputs: [10586(1525KB)], [10584(15MB)]
>
>2018-07-11 11:58:40.509906 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320509896, "job": 973, "event": "compaction_started", "files_L0": [10586], "files_L1": [10584], "score": -1, "input_data_size": 18215569}
>2018-07-11 11:58:40.718148 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 973] Generated table #10587: 3653 keys, 17606923 bytes
>2018-07-11 11:58:40.718204 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320718189, "cf_name": "default", "job": 973, "event": "table_file_creation", "file_number": 10587, "file_size": 17606923, "table_properties": {"data_size": 17474142, "index_size": 75406, "filter_size": 56448, "raw_key_size": 73585, "raw_average_key_size": 20, "raw_value_size": 17369399, "raw_average_value_size": 4754, "num_data_blocks": 2203, "num_entries": 3653, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 11:58:40.751246 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 973] Compacted 1@0 + 1@1 files to L1 => 17606923 bytes
>2018-07-11 11:58:40.793172 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.793106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 75.5 rd, 73.0 wr, level 1, files in(1, 1) out(1) MB in(1.5, 15.9) out(16.8), read-write-amplify(22.9) write-amplify(11.3) OK, records in: 3999, records dropped: 346
>
>2018-07-11 11:58:40.793190 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-11:58:40.793127) EVENT_LOG_v1 {"time_micros": 1531310320793118, "job": 973, "event": "compaction_finished", "compaction_time_micros": 241322, "output_level": 1, "num_output_files": 1, "total_output_size": 17606923, "num_input_records": 3999, "num_output_records": 3653, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 11:58:40.793569 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320793565, "job": 973, "event": "table_file_deletion", "file_number": 10586}
>2018-07-11 11:58:40.796500 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310320796497, "job": 973, "event": "table_file_deletion", "file_number": 10584}
>2018-07-11 11:58:40.796575 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 11:59:03.338135 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 11:59:07.304542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:59:07.472124 7f29149e4700  1 mon.magna102@0(leader).log v21246 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21246)
>2018-07-11 11:59:07.472170 7f29149e4700  1 mon.magna102@0(leader).log v21246 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21246)
>2018-07-11 11:59:19.309864 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 11:59:19.481520 7f29149e4700  1 mon.magna102@0(leader).log v21247 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21247)
>2018-07-11 11:59:19.481566 7f29149e4700  1 mon.magna102@0(leader).log v21247 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21247)
>2018-07-11 11:59:29.314159 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 11:59:29.490726 7f29149e4700  1 mon.magna102@0(leader).log v21248 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21248)
>2018-07-11 11:59:29.490772 7f29149e4700  1 mon.magna102@0(leader).log v21248 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21248)
>2018-07-11 11:59:35.360370 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 11:59:35.474689 7f29149e4700  1 mon.magna102@0(leader).log v21249 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21249)
>2018-07-11 11:59:35.474735 7f29149e4700  1 mon.magna102@0(leader).log v21249 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21249)
>2018-07-11 12:00:03.361269 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 12:00:47.347961 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:00:47.514817 7f29149e4700  1 mon.magna102@0(leader).log v21250 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21250)
>2018-07-11 12:00:47.514863 7f29149e4700  1 mon.magna102@0(leader).log v21250 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21250)
>2018-07-11 12:00:49.515327 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10588. Immutable memtables: 0.
>
>2018-07-11 12:00:49.515370 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 974] Syncing log #10585
>2018-07-11 12:00:49.531907 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.515361) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:00:49.531919 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 974] Flushing memtable with next log file: 10588
>
>2018-07-11 12:00:49.531933 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449531925, "job": 974, "event": "flush_started", "num_memtables": 1, "num_entries": 751, "num_deletes": 251, "memory_usage": 1119896}
>2018-07-11 12:00:49.531936 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 974] Level-0 flush table #10589: started
>2018-07-11 12:00:49.582438 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449582424, "cf_name": "default", "job": 974, "event": "table_file_creation", "file_number": 10589, "file_size": 1060419, "table_properties": {"data_size": 1050867, "index_size": 4656, "filter_size": 3976, "raw_key_size": 8282, "raw_average_key_size": 20, "raw_value_size": 1041956, "raw_average_value_size": 2572, "num_data_blocks": 142, "num_entries": 405, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:00:49.582448 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 974] Level-0 flush table #10589: 1060419 bytes OK
>2018-07-11 12:00:49.632504 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.599002) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10589 started
>2018-07-11 12:00:49.632512 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.632468) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10589: memtable #1 done
>2018-07-11 12:00:49.632515 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.632482) EVENT_LOG_v1 {"time_micros": 1531310449632476, "job": 974, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:00:49.632517 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.632494) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:00:49.632523 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 974] Try to delete WAL files size 1103586, prev total WAL file size 1103586, number of live WAL files 2.
>
>2018-07-11 12:00:49.632826 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:00:49.632880 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.632873) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:00:49.632889 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 975] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:00:49.632895 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 974 Base level 0, inputs: [10589(1035KB)], [10587(16MB)]
>
>2018-07-11 12:00:49.632908 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449632898, "job": 975, "event": "compaction_started", "files_L0": [10589], "files_L1": [10587], "score": -1, "input_data_size": 18667342}
>2018-07-11 12:00:49.851113 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 975] Generated table #10590: 3548 keys, 16687892 bytes
>2018-07-11 12:00:49.851162 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449851134, "cf_name": "default", "job": 975, "event": "table_file_creation", "file_number": 10590, "file_size": 16687892, "table_properties": {"data_size": 16561344, "index_size": 71935, "filter_size": 53686, "raw_key_size": 71620, "raw_average_key_size": 20, "raw_value_size": 16460460, "raw_average_value_size": 4639, "num_data_blocks": 2093, "num_entries": 3548, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:00:49.884294 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 975] Compacted 1@0 + 1@1 files to L1 => 16687892 bytes
>2018-07-11 12:00:49.917819 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.917744) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.3 rd, 66.4 wr, level 1, files in(1, 1) out(1) MB in(1.0, 16.8) out(15.9), read-write-amplify(33.3) write-amplify(15.7) OK, records in: 4058, records dropped: 510
>
>2018-07-11 12:00:49.917828 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:00:49.917766) EVENT_LOG_v1 {"time_micros": 1531310449917756, "job": 975, "event": "compaction_finished", "compaction_time_micros": 251369, "output_level": 1, "num_output_files": 1, "total_output_size": 16687892, "num_input_records": 4058, "num_output_records": 3548, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:00:49.918093 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449918088, "job": 975, "event": "table_file_deletion", "file_number": 10589}
>2018-07-11 12:00:49.921317 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310449921314, "job": 975, "event": "table_file_deletion", "file_number": 10587}
>2018-07-11 12:00:59.353514 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:00:59.510689 7f29149e4700  1 mon.magna102@0(leader).log v21251 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21251)
>2018-07-11 12:00:59.510734 7f29149e4700  1 mon.magna102@0(leader).log v21251 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21251)
>2018-07-11 12:01:03.385975 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:01:13.359535 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 12:01:13.512036 7f29149e4700  1 mon.magna102@0(leader).log v21252 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21252)
>2018-07-11 12:01:13.512082 7f29149e4700  1 mon.magna102@0(leader).log v21252 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21252)
>2018-07-11 12:01:20.395534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:01:20.495923 7f29149e4700  1 mon.magna102@0(leader).log v21253 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21253)
>2018-07-11 12:01:20.495968 7f29149e4700  1 mon.magna102@0(leader).log v21253 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21253)
>2018-07-11 12:02:03.416773 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:02:33.394259 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:02:33.569273 7f29149e4700  1 mon.magna102@0(leader).log v21254 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21254)
>2018-07-11 12:02:33.569319 7f29149e4700  1 mon.magna102@0(leader).log v21254 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21254)
>2018-07-11 12:02:39.397007 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:02:39.569853 7f29149e4700  1 mon.magna102@0(leader).log v21255 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21255)
>2018-07-11 12:02:39.569899 7f29149e4700  1 mon.magna102@0(leader).log v21255 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21255)
>2018-07-11 12:02:53.402813 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 12:02:53.545970 7f29149e4700  1 mon.magna102@0(leader).log v21256 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21256)
>2018-07-11 12:02:53.546016 7f29149e4700  1 mon.magna102@0(leader).log v21256 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21256)
>2018-07-11 12:03:00.437788 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:03:00.539470 7f29149e4700  1 mon.magna102@0(leader).log v21257 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21257)
>2018-07-11 12:03:00.539515 7f29149e4700  1 mon.magna102@0(leader).log v21257 check_sub sending message to client.? 10.8.128.102:0/831730969 with 1 entries (version 21257)
>2018-07-11 12:03:03.439621 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:04:03.460486 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:04:07.435120 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:04:07.594412 7f29149e4700  1 mon.magna102@0(leader).log v21258 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21258)
>2018-07-11 12:04:19.440617 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:04:19.613685 7f29149e4700  1 mon.magna102@0(leader).log v21259 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21259)
>2018-07-11 12:04:37.448243 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 12:04:37.598818 7f29149e4700  1 mon.magna102@0(leader).log v21260 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21260)
>2018-07-11 12:04:45.482806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:04:45.599825 7f29149e4700  1 mon.magna102@0(leader).log v21261 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21261)
>2018-07-11 12:05:03.492635 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:05:14.398385 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:05:14.398406 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3792523464' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:05:14.594860 7f29149e4700  1 mon.magna102@0(leader).log v21262 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21262)
>2018-07-11 12:05:47.478474 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:05:47.647403 7f29149e4700  1 mon.magna102@0(leader).log v21263 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21263)
>2018-07-11 12:05:59.483646 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:05:59.656668 7f29149e4700  1 mon.magna102@0(leader).log v21264 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21264)
>2018-07-11 12:06:03.523426 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:06:15.490593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:06:15.667594 7f29149e4700  1 mon.magna102@0(leader).log v21265 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21265)
>2018-07-11 12:06:20.532908 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:06:20.634722 7f29149e4700  1 mon.magna102@0(leader).log v21266 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21266)
>2018-07-11 12:07:00.613085 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10591. Immutable memtables: 0.
>
>2018-07-11 12:07:00.613127 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 976] Syncing log #10588
>2018-07-11 12:07:00.629481 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.613117) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:07:00.629492 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 976] Flushing memtable with next log file: 10591
>
>2018-07-11 12:07:00.629519 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310820629510, "job": 976, "event": "flush_started", "num_memtables": 1, "num_entries": 1678, "num_deletes": 250, "memory_usage": 3179672}
>2018-07-11 12:07:00.629522 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 976] Level-0 flush table #10592: started
>2018-07-11 12:07:00.680054 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310820680040, "cf_name": "default", "job": 976, "event": "table_file_creation", "file_number": 10592, "file_size": 1796940, "table_properties": {"data_size": 1782596, "index_size": 7241, "filter_size": 6183, "raw_key_size": 10537, "raw_average_key_size": 21, "raw_value_size": 1770512, "raw_average_value_size": 3555, "num_data_blocks": 226, "num_entries": 498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:07:00.680065 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 976] Level-0 flush table #10592: 1796940 bytes OK
>2018-07-11 12:07:00.738401 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.696576) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10592 started
>2018-07-11 12:07:00.738409 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.738364) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10592: memtable #1 done
>2018-07-11 12:07:00.738412 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.738378) EVENT_LOG_v1 {"time_micros": 1531310820738372, "job": 976, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:07:00.738426 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.738391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:07:00.738433 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 976] Try to delete WAL files size 3147478, prev total WAL file size 3147478, number of live WAL files 2.
>
>2018-07-11 12:07:00.739206 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:07:00.739388 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 12:07:00.739395 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 72847.6 total, 837.3 interval
>Cumulative writes: 97K writes, 461K keys, 97K commit groups, 1.0 writes per commit group, ingest: 0.77 GB, 0.01 MB/s
>Cumulative WAL: 97K writes, 97K syncs, 1.00 writes per sync, written: 0.77 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 906 writes, 3948 keys, 906 commit groups, 1.0 writes per commit group, ingest: 6.71 MB, 0.01 MB/s
>Interval WAL: 906 writes, 906 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       489    0.081       0      0
>  L1      1/1   15.91 MB   0.0      7.7     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       109       486    0.223   1746K   209K
> Sum      2/2   17.63 MB   0.0      7.7     0.7      7.0       7.7      0.7       0.0  11.6     53.4     53.4       148       975    0.152   1746K   209K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.0       0.1      0.0       0.0  12.5     57.6     57.5         1         6    0.153     12K   1366
>Uptime(secs): 72847.6 total, 72847.6 interval
>Flush(GB): cumulative 0.662, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.71 GB write, 0.11 MB/s write, 7.71 GB read, 0.11 MB/s read, 147.9 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        39       489    0.081       0      0
>  L1      1/1   15.91 MB   0.0      7.7     0.7      7.0       7.0      0.0       0.0  10.7     72.7     66.5       109       486    0.223   1746K   209K
> Sum      2/2   17.63 MB   0.0      7.7     0.7      7.0       7.7      0.7       0.0  11.6     53.4     53.4       148       975    0.152   1746K   209K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 72847.6 total, 72847.6 interval
>Flush(GB): cumulative 0.662, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.71 GB write, 0.11 MB/s write, 7.71 GB read, 0.11 MB/s read, 147.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 12:07:00.739416 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:07:00.739410) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:07:00.739421 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 977] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:07:00.739426 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 976 Base level 0, inputs: [10592(1754KB)], [10590(15MB)]
>
>2018-07-11 12:07:00.739439 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310820739430, "job": 977, "event": "compaction_started", "files_L0": [10592], "files_L1": [10590], "score": -1, "input_data_size": 18484832}
>2018-07-11 12:07:00.947507 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 977] Generated table #10593: 3717 keys, 17992543 bytes
>2018-07-11 12:07:00.947556 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310820947541, "cf_name": "default", "job": 977, "event": "table_file_creation", "file_number": 10593, "file_size": 17992543, "table_properties": {"data_size": 17857064, "index_size": 76915, "filter_size": 57637, "raw_key_size": 74886, "raw_average_key_size": 20, "raw_value_size": 17750358, "raw_average_value_size": 4775, "num_data_blocks": 2247, "num_entries": 3717, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:07:00.980625 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 977] Compacted 1@0 + 1@1 files to L1 => 17992543 bytes
>2018-07-11 12:07:01.014145 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:07:01.014077) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 76.6 rd, 74.6 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.9) out(17.2), read-write-amplify(20.3) write-amplify(10.0) OK, records in: 4046, records dropped: 329
>
>2018-07-11 12:07:01.014154 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:07:01.014098) EVENT_LOG_v1 {"time_micros": 1531310821014088, "job": 977, "event": "compaction_finished", "compaction_time_micros": 241172, "output_level": 1, "num_output_files": 1, "total_output_size": 17992543, "num_input_records": 4046, "num_output_records": 3717, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:07:01.014590 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310821014586, "job": 977, "event": "table_file_deletion", "file_number": 10592}
>2018-07-11 12:07:01.017514 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310821017511, "job": 977, "event": "table_file_deletion", "file_number": 10590}
>2018-07-11 12:07:01.017577 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:07:03.548114 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 12:07:33.524361 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:07:33.676833 7f29149e4700  1 mon.magna102@0(leader).log v21267 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21267)
>2018-07-11 12:07:39.527292 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:07:39.677375 7f29149e4700  1 mon.magna102@0(leader).log v21268 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21268)
>2018-07-11 12:07:53.532954 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 12:07:53.695251 7f29149e4700  1 mon.magna102@0(leader).log v21269 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21269)
>2018-07-11 12:08:00.587065 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:08:00.687486 7f29149e4700  1 mon.magna102@0(leader).log v21270 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21270)
>2018-07-11 12:08:03.579307 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 12:08:05.688596 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10594. Immutable memtables: 0.
>
>2018-07-11 12:08:05.688639 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 978] Syncing log #10591
>2018-07-11 12:08:05.704911 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.688630) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:08:05.704922 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 978] Flushing memtable with next log file: 10594
>
>2018-07-11 12:08:05.704949 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310885704941, "job": 978, "event": "flush_started", "num_memtables": 1, "num_entries": 768, "num_deletes": 500, "memory_usage": 625112}
>2018-07-11 12:08:05.704952 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 978] Level-0 flush table #10595: started
>2018-07-11 12:08:05.755459 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310885755445, "cf_name": "default", "job": 978, "event": "table_file_creation", "file_number": 10595, "file_size": 586653, "table_properties": {"data_size": 580103, "index_size": 2544, "filter_size": 3087, "raw_key_size": 12106, "raw_average_key_size": 20, "raw_value_size": 570410, "raw_average_value_size": 971, "num_data_blocks": 78, "num_entries": 587, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 12:08:05.755469 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 978] Level-0 flush table #10595: 586653 bytes OK
>2018-07-11 12:08:05.805486 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.771959) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10595 started
>2018-07-11 12:08:05.805495 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.805450) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10595: memtable #1 done
>2018-07-11 12:08:05.805497 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.805465) EVENT_LOG_v1 {"time_micros": 1531310885805458, "job": 978, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:08:05.805499 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.805477) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:08:05.805519 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 978] Try to delete WAL files size 606287, prev total WAL file size 606287, number of live WAL files 2.
>
>2018-07-11 12:08:05.805807 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:08:05.805860 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:05.805852) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 12:08:05.805869 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 979] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:08:05.805875 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 978 Base level 0, inputs: [10595(572KB)], [10593(17MB)]
>
>2018-07-11 12:08:05.805889 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310885805879, "job": 979, "event": "compaction_started", "files_L0": [10595], "files_L1": [10593], "score": -1, "input_data_size": 18579196}
>2018-07-11 12:08:06.015729 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 979] Generated table #10596: 3296 keys, 16139795 bytes
>2018-07-11 12:08:06.015778 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310886015750, "cf_name": "default", "job": 979, "event": "table_file_creation", "file_number": 10596, "file_size": 16139795, "table_properties": {"data_size": 16017004, "index_size": 70028, "filter_size": 51836, "raw_key_size": 66286, "raw_average_key_size": 20, "raw_value_size": 15921116, "raw_average_value_size": 4830, "num_data_blocks": 2057, "num_entries": 3296, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:08:06.048955 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 979] Compacted 1@0 + 1@1 files to L1 => 16139795 bytes
>2018-07-11 12:08:06.082451 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:06.082382) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 66.4 wr, level 1, files in(1, 1) out(1) MB in(0.6, 17.2) out(15.4), read-write-amplify(59.2) write-amplify(27.5) OK, records in: 4304, records dropped: 1008
>
>2018-07-11 12:08:06.082460 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:06.082404) EVENT_LOG_v1 {"time_micros": 1531310886082394, "job": 979, "event": "compaction_finished", "compaction_time_micros": 243049, "output_level": 1, "num_output_files": 1, "total_output_size": 16139795, "num_input_records": 4304, "num_output_records": 3296, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:08:06.082660 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310886082656, "job": 979, "event": "table_file_deletion", "file_number": 10595}
>2018-07-11 12:08:06.086157 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310886086154, "job": 979, "event": "table_file_deletion", "file_number": 10593}
>2018-07-11 12:08:06.086210 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:08:07.578301 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:08:07.578323 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/70986760' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:08:07.691588 7f29149e4700  1 mon.magna102@0(leader).log v21271 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21271)
>2018-07-11 12:08:08.986847 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:08:08.986865 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/1538304589' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:08:09.116755 7f29149e4700  1 mon.magna102@0(leader).log v21272 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21272)
>2018-07-11 12:08:10.286111 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:08:10.286130 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3633477357' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:08:11.275477 7f29149e4700  1 mon.magna102@0(leader).log v21273 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21273)
>2018-07-11 12:08:19.676685 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10597. Immutable memtables: 0.
>
>2018-07-11 12:08:19.676730 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 980] Syncing log #10594
>2018-07-11 12:08:19.693277 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.676720) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:08:19.693288 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 980] Flushing memtable with next log file: 10597
>
>2018-07-11 12:08:19.693315 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310899693306, "job": 980, "event": "flush_started", "num_memtables": 1, "num_entries": 329, "num_deletes": 251, "memory_usage": 188544}
>2018-07-11 12:08:19.693318 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 980] Level-0 flush table #10598: started
>2018-07-11 12:08:19.727110 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310899727095, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 10598, "file_size": 176782, "table_properties": {"data_size": 173982, "index_size": 706, "filter_size": 1175, "raw_key_size": 5724, "raw_average_key_size": 20, "raw_value_size": 169657, "raw_average_value_size": 599, "num_data_blocks": 22, "num_entries": 283, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:08:19.727120 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 980] Level-0 flush table #10598: 176782 bytes OK
>2018-07-11 12:08:19.785501 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.743676) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10598 started
>2018-07-11 12:08:19.785509 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.785464) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10598: memtable #1 done
>2018-07-11 12:08:19.785512 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.785479) EVENT_LOG_v1 {"time_micros": 1531310899785473, "job": 980, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:08:19.785527 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.785492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:08:19.785533 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 980] Try to delete WAL files size 179610, prev total WAL file size 179610, number of live WAL files 2.
>
>2018-07-11 12:08:19.785668 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:08:19.785718 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:19.785711) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:08:19.785728 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 981] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:08:19.785734 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 980 Base level 0, inputs: [10598(172KB)], [10596(15MB)]
>
>2018-07-11 12:08:19.785747 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310899785737, "job": 981, "event": "compaction_started", "files_L0": [10598], "files_L1": [10596], "score": -1, "input_data_size": 16316577}
>2018-07-11 12:08:19.952878 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 981] Generated table #10599: 3069 keys, 14350976 bytes
>2018-07-11 12:08:19.952927 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310899952900, "cf_name": "default", "job": 981, "event": "table_file_creation", "file_number": 10599, "file_size": 14350976, "table_properties": {"data_size": 14241262, "index_size": 62486, "filter_size": 46301, "raw_key_size": 61763, "raw_average_key_size": 20, "raw_value_size": 14153823, "raw_average_value_size": 4611, "num_data_blocks": 1827, "num_entries": 3069, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:08:19.986050 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 981] Compacted 1@0 + 1@1 files to L1 => 14350976 bytes
>2018-07-11 12:08:20.019596 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:20.019526) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 81.5 rd, 71.7 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.4) out(13.7), read-write-amplify(173.5) write-amplify(81.2) OK, records in: 3579, records dropped: 510
>
>2018-07-11 12:08:20.019605 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:08:20.019550) EVENT_LOG_v1 {"time_micros": 1531310900019538, "job": 981, "event": "compaction_finished", "compaction_time_micros": 200289, "output_level": 1, "num_output_files": 1, "total_output_size": 14350976, "num_input_records": 3579, "num_output_records": 3069, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:08:20.019726 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310900019721, "job": 981, "event": "table_file_deletion", "file_number": 10598}
>2018-07-11 12:08:20.022689 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531310900022686, "job": 981, "event": "table_file_deletion", "file_number": 10596}
>2018-07-11 12:09:03.607559 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:09:13.568630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:09:13.733581 7f29149e4700  1 mon.magna102@0(leader).log v21274 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21274)
>2018-07-11 12:09:19.571541 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:09:19.717501 7f29149e4700  1 mon.magna102@0(leader).log v21275 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21275)
>2018-07-11 12:09:37.580585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 12:09:37.728676 7f29149e4700  1 mon.magna102@0(leader).log v21276 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21276)
>2018-07-11 12:09:45.621871 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:09:45.738904 7f29149e4700  1 mon.magna102@0(leader).log v21277 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21277)
>2018-07-11 12:10:03.632304 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:10:04.659874 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:10:04.659901 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4188151159' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:10:04.661082 7f29191ed700  1 mon.magna102@0(leader).log v21277 check_sub sending message to client.? 10.8.128.102:0/4188151159 with 1 entries (version 21277)
>2018-07-11 12:10:04.749114 7f29149e4700  1 mon.magna102@0(leader).log v21278 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21278)
>2018-07-11 12:10:04.749146 7f29149e4700  1 mon.magna102@0(leader).log v21278 check_sub sending message to client.? 10.8.128.102:0/4188151159 with 0 entries (version 21278)
>2018-07-11 12:10:13.250525 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:10:13.250550 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/359522623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:10:13.366627 7f29149e4700  1 mon.magna102@0(leader).log v21279 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21279)
>2018-07-11 12:10:18.560216 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 12:10:18.560244 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2668242726' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 12:10:18.758920 7f29149e4700  1 mon.magna102@0(leader).log v21280 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21280)
>2018-07-11 12:10:22.665813 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd lspools"} v 0) v1
>2018-07-11 12:10:22.665841 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3405583258' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch
>2018-07-11 12:10:22.767749 7f29149e4700  1 mon.magna102@0(leader).log v21281 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21281)
>2018-07-11 12:10:32.920529 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:10:32.920553 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1652865291' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:10:33.060526 7f29149e4700  1 mon.magna102@0(leader).log v21282 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21282)
>2018-07-11 12:10:53.616265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:10:53.763550 7f29149e4700  1 mon.magna102@0(leader).log v21283 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21283)
>2018-07-11 12:11:03.647331 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:11:05.621840 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:11:05.772832 7f29149e4700  1 mon.magna102@0(leader).log v21284 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21284)
>2018-07-11 12:11:15.626279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:11:15.782022 7f29149e4700  1 mon.magna102@0(leader).log v21285 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21285)
>2018-07-11 12:11:20.670633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,5 (REQUEST_SLOW)
>2018-07-11 12:11:20.765796 7f29149e4700  1 mon.magna102@0(leader).log v21286 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21286)
>2018-07-11 12:12:03.647498 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:12:03.647625 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 12:12:03.803724 7f29149e4700  1 mon.magna102@0(leader).log v21287 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21287)
>2018-07-11 12:12:09.037465 7f29149e4700  1 mon.magna102@0(leader).log v21288 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21288)
>2018-07-11 12:12:10.037501 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 12:12:10.187838 7f29149e4700  1 mon.magna102@0(leader).log v21289 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21289)
>2018-07-11 12:12:45.705628 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:12:45.705652 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1279828521' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:12:45.823807 7f29149e4700  1 mon.magna102@0(leader).log v21290 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21290)
>2018-07-11 12:12:48.304797 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:12:48.304823 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2359830125' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:12:48.306115 7f29191ed700  1 mon.magna102@0(leader).log v21290 check_sub sending message to client.? 10.8.128.102:0/2359830125 with 1 entries (version 21290)
>2018-07-11 12:12:48.832481 7f29149e4700  1 mon.magna102@0(leader).log v21291 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21291)
>2018-07-11 12:12:48.832501 7f29149e4700  1 mon.magna102@0(leader).log v21291 check_sub sending message to client.? 10.8.128.102:0/2359830125 with 0 entries (version 21291)
>2018-07-11 12:12:57.889523 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 12:12:57.889551 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3373528653' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 12:12:58.809603 7f29149e4700  1 mon.magna102@0(leader).log v21292 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21292)
>2018-07-11 12:13:00.173664 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:13:00.173687 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1421614850' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:13:00.943305 7f29149e4700  1 mon.magna102@0(leader).log v21293 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21293)
>2018-07-11 12:13:03.647684 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:14:02.813351 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:14:02.813375 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3234086185' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:14:02.932437 7f29149e4700  1 mon.magna102@0(leader).log v21294 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21294)
>2018-07-11 12:14:03.647870 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:14:06.624284 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:14:06.624310 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/626313257' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:14:06.625631 7f29191ed700  1 mon.magna102@0(leader).log v21294 check_sub sending message to client.? 10.8.128.102:0/626313257 with 1 entries (version 21294)
>2018-07-11 12:14:06.832913 7f29149e4700  1 mon.magna102@0(leader).log v21295 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21295)
>2018-07-11 12:14:06.832946 7f29149e4700  1 mon.magna102@0(leader).log v21295 check_sub sending message to client.? 10.8.128.102:0/626313257 with 0 entries (version 21295)
>2018-07-11 12:14:50.468886 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:14:50.468908 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3632056405' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:14:50.860978 7f29149e4700  1 mon.magna102@0(leader).log v21296 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21296)
>2018-07-11 12:15:03.648031 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:15:20.856954 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10600. Immutable memtables: 0.
>
>2018-07-11 12:15:20.857008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 982] Syncing log #10597
>2018-07-11 12:15:20.873015 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.856999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:15:20.873026 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 982] Flushing memtable with next log file: 10600
>
>2018-07-11 12:15:20.873053 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311320873045, "job": 982, "event": "flush_started", "num_memtables": 1, "num_entries": 1909, "num_deletes": 250, "memory_usage": 3671816}
>2018-07-11 12:15:20.873056 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 982] Level-0 flush table #10601: started
>2018-07-11 12:15:20.931913 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311320931898, "cf_name": "default", "job": 982, "event": "table_file_creation", "file_number": 10601, "file_size": 2112173, "table_properties": {"data_size": 2095587, "index_size": 8507, "filter_size": 7158, "raw_key_size": 11464, "raw_average_key_size": 21, "raw_value_size": 2081942, "raw_average_value_size": 3827, "num_data_blocks": 265, "num_entries": 544, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:15:20.931923 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 982] Level-0 flush table #10601: 2112173 bytes OK
>2018-07-11 12:15:20.990280 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.948435) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10601 started
>2018-07-11 12:15:20.990288 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.990244) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10601: memtable #1 done
>2018-07-11 12:15:20.990291 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.990259) EVENT_LOG_v1 {"time_micros": 1531311320990253, "job": 982, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:15:20.990293 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.990271) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:15:20.990312 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 982] Try to delete WAL files size 3635439, prev total WAL file size 3635439, number of live WAL files 2.
>
>2018-07-11 12:15:20.991191 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:15:20.991257 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:20.991249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:15:20.991267 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 983] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:15:20.991274 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 982 Base level 0, inputs: [10601(2062KB)], [10599(13MB)]
>
>2018-07-11 12:15:20.991288 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311320991278, "job": 983, "event": "compaction_started", "files_L0": [10601], "files_L1": [10599], "score": -1, "input_data_size": 16463149}
>2018-07-11 12:15:21.174298 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 983] Generated table #10602: 3309 keys, 16142341 bytes
>2018-07-11 12:15:21.174344 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311321174328, "cf_name": "default", "job": 983, "event": "table_file_creation", "file_number": 10602, "file_size": 16142341, "table_properties": {"data_size": 16019982, "index_size": 69643, "filter_size": 51789, "raw_key_size": 66506, "raw_average_key_size": 20, "raw_value_size": 15924189, "raw_average_value_size": 4812, "num_data_blocks": 2045, "num_entries": 3309, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:15:21.207508 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 983] Compacted 1@0 + 1@1 files to L1 => 16142341 bytes
>2018-07-11 12:15:21.249352 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:21.249280) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.1 rd, 74.7 wr, level 1, files in(1, 1) out(1) MB in(2.0, 13.7) out(15.4), read-write-amplify(15.4) write-amplify(7.6) OK, records in: 3613, records dropped: 304
>
>2018-07-11 12:15:21.249361 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:21.249301) EVENT_LOG_v1 {"time_micros": 1531311321249292, "job": 983, "event": "compaction_finished", "compaction_time_micros": 216204, "output_level": 1, "num_output_files": 1, "total_output_size": 16142341, "num_input_records": 3613, "num_output_records": 3309, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:15:21.249849 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311321249845, "job": 983, "event": "table_file_deletion", "file_number": 10601}
>2018-07-11 12:15:21.252721 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311321252718, "job": 983, "event": "table_file_deletion", "file_number": 10599}
>2018-07-11 12:15:21.252839 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:15:51.910819 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10603. Immutable memtables: 0.
>
>2018-07-11 12:15:51.910861 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 984] Syncing log #10600
>2018-07-11 12:15:51.927447 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:51.910851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:15:51.927458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 984] Flushing memtable with next log file: 10603
>
>2018-07-11 12:15:51.927484 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311351927476, "job": 984, "event": "flush_started", "num_memtables": 1, "num_entries": 368, "num_deletes": 251, "memory_usage": 259304}
>2018-07-11 12:15:51.927487 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 984] Level-0 flush table #10604: started
>2018-07-11 12:15:51.969601 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311351969589, "cf_name": "default", "job": 984, "event": "table_file_creation", "file_number": 10604, "file_size": 243356, "table_properties": {"data_size": 239925, "index_size": 1102, "filter_size": 1410, "raw_key_size": 5868, "raw_average_key_size": 20, "raw_value_size": 235246, "raw_average_value_size": 811, "num_data_blocks": 34, "num_entries": 290, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:15:51.969612 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 984] Level-0 flush table #10604: 243356 bytes OK
>2018-07-11 12:15:52.019629 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:51.986145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10604 started
>2018-07-11 12:15:52.019637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.019594) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10604: memtable #1 done
>2018-07-11 12:15:52.019639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.019608) EVENT_LOG_v1 {"time_micros": 1531311352019602, "job": 984, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:15:52.019641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.019620) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:15:52.019659 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 984] Try to delete WAL files size 249929, prev total WAL file size 249929, number of live WAL files 2.
>
>2018-07-11 12:15:52.019795 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:15:52.019844 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.019837) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:15:52.019854 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 985] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:15:52.019860 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 984 Base level 0, inputs: [10604(237KB)], [10602(15MB)]
>
>2018-07-11 12:15:52.019873 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311352019863, "job": 985, "event": "compaction_started", "files_L0": [10604], "files_L1": [10602], "score": -1, "input_data_size": 16385697}
>2018-07-11 12:15:52.196574 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 985] Generated table #10605: 3091 keys, 14412023 bytes
>2018-07-11 12:15:52.196617 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311352196590, "cf_name": "default", "job": 985, "event": "table_file_creation", "file_number": 10605, "file_size": 14412023, "table_properties": {"data_size": 14302118, "index_size": 62507, "filter_size": 46471, "raw_key_size": 62178, "raw_average_key_size": 20, "raw_value_size": 14214359, "raw_average_value_size": 4598, "num_data_blocks": 1827, "num_entries": 3091, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:15:52.238116 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 985] Compacted 1@0 + 1@1 files to L1 => 14412023 bytes
>2018-07-11 12:15:52.271506 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.271437) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.1 rd, 66.0 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.4) out(13.7), read-write-amplify(126.6) write-amplify(59.2) OK, records in: 3599, records dropped: 508
>
>2018-07-11 12:15:52.271515 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:15:52.271459) EVENT_LOG_v1 {"time_micros": 1531311352271450, "job": 985, "event": "compaction_finished", "compaction_time_micros": 218229, "output_level": 1, "num_output_files": 1, "total_output_size": 14412023, "num_input_records": 3599, "num_output_records": 3091, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:15:52.271645 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311352271641, "job": 985, "event": "table_file_deletion", "file_number": 10604}
>2018-07-11 12:15:52.274617 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311352274614, "job": 985, "event": "table_file_deletion", "file_number": 10602}
>2018-07-11 12:16:03.648173 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:17:03.648373 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:18:03.648546 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:19:03.648719 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:20:03.648877 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:20:21.417797 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:20:21.417819 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3491307625' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:20:21.536399 7f29149e4700  1 mon.magna102@0(leader).log v21297 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21297)
>2018-07-11 12:20:25.120678 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:20:25.120703 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/1915903261' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:20:25.122057 7f29191ed700  1 mon.magna102@0(leader).log v21297 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 1 entries (version 21297)
>2018-07-11 12:20:25.236889 7f29149e4700  1 mon.magna102@0(leader).log v21298 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21298)
>2018-07-11 12:20:25.236922 7f29149e4700  1 mon.magna102@0(leader).log v21298 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 0 entries (version 21298)
>2018-07-11 12:21:03.649042 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:21:09.049921 7f29149e4700  1 mon.magna102@0(leader).log v21299 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21299)
>2018-07-11 12:21:09.049955 7f29149e4700  1 mon.magna102@0(leader).log v21299 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 0 entries (version 21299)
>2018-07-11 12:21:11.216897 7f29149e4700  1 mon.magna102@0(leader).log v21300 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21300)
>2018-07-11 12:21:11.216919 7f29149e4700  1 mon.magna102@0(leader).log v21300 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 0 entries (version 21300)
>2018-07-11 12:21:37.385491 7f29149e4700  1 mon.magna102@0(leader).log v21301 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21301)
>2018-07-11 12:21:37.385523 7f29149e4700  1 mon.magna102@0(leader).log v21301 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 0 entries (version 21301)
>2018-07-11 12:21:47.370945 7f29149e4700  1 mon.magna102@0(leader).log v21302 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21302)
>2018-07-11 12:21:47.370978 7f29149e4700  1 mon.magna102@0(leader).log v21302 check_sub sending message to client.? 10.8.128.107:0/1915903261 with 0 entries (version 21302)
>2018-07-11 12:22:03.649218 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:23:03.649390 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:23:41.047114 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10606. Immutable memtables: 0.
>
>2018-07-11 12:23:41.047156 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 986] Syncing log #10603
>2018-07-11 12:23:41.063570 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.047147) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:23:41.063582 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 986] Flushing memtable with next log file: 10606
>
>2018-07-11 12:23:41.063608 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821063600, "job": 986, "event": "flush_started", "num_memtables": 1, "num_entries": 1948, "num_deletes": 250, "memory_usage": 3646352}
>2018-07-11 12:23:41.063611 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 986] Level-0 flush table #10607: started
>2018-07-11 12:23:41.122457 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821122443, "cf_name": "default", "job": 986, "event": "table_file_creation", "file_number": 10607, "file_size": 1930018, "table_properties": {"data_size": 1914314, "index_size": 8108, "filter_size": 6676, "raw_key_size": 10997, "raw_average_key_size": 21, "raw_value_size": 1901266, "raw_average_value_size": 3649, "num_data_blocks": 254, "num_entries": 521, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:23:41.122468 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 986] Level-0 flush table #10607: 1930018 bytes OK
>2018-07-11 12:23:41.172469 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.138947) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10607 started
>2018-07-11 12:23:41.172477 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.172432) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10607: memtable #1 done
>2018-07-11 12:23:41.172479 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.172447) EVENT_LOG_v1 {"time_micros": 1531311821172441, "job": 986, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:23:41.172494 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.172459) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:23:41.172501 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 986] Try to delete WAL files size 3609335, prev total WAL file size 3609335, number of live WAL files 2.
>
>2018-07-11 12:23:41.173362 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:23:41.173545 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 12:23:41.173552 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 73848.0 total, 1000.4 interval
>Cumulative writes: 98K writes, 466K keys, 98K commit groups, 1.0 writes per commit group, ingest: 0.77 GB, 0.01 MB/s
>Cumulative WAL: 98K writes, 98K syncs, 1.00 writes per sync, written: 0.77 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1084 writes, 5322 keys, 1084 commit groups, 1.0 writes per commit group, ingest: 7.90 MB, 0.01 MB/s
>Interval WAL: 1084 writes, 1084 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.84 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       494    0.080       0      0
>  L1      1/1   13.74 MB   0.0      7.8     0.7      7.1       7.1     -0.0       0.0  10.7     72.7     66.5       110       491    0.223   1765K   212K
> Sum      2/2   15.58 MB   0.0      7.8     0.7      7.1       7.8      0.7       0.0  11.7     53.4     53.4       149       985    0.152   1765K   212K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.7     56.9     55.5         1        10    0.145     19K   2659
>Uptime(secs): 73848.0 total, 73848.0 interval
>Flush(GB): cumulative 0.667, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.79 GB write, 0.11 MB/s write, 7.79 GB read, 0.11 MB/s read, 149.3 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.4 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.84 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       494    0.080       0      0
>  L1      1/1   13.74 MB   0.0      7.8     0.7      7.1       7.1     -0.0       0.0  10.7     72.7     66.5       110       491    0.223   1765K   212K
> Sum      2/2   15.58 MB   0.0      7.8     0.7      7.1       7.8      0.7       0.0  11.7     53.4     53.4       149       985    0.152   1765K   212K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 73848.0 total, 73848.0 interval
>Flush(GB): cumulative 0.667, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.79 GB write, 0.11 MB/s write, 7.79 GB read, 0.11 MB/s read, 149.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 12:23:41.173573 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.173567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:23:41.173578 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 987] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:23:41.173583 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 986 Base level 0, inputs: [10607(1884KB)], [10605(13MB)]
>
>2018-07-11 12:23:41.173596 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821173586, "job": 987, "event": "compaction_started", "files_L0": [10607], "files_L1": [10605], "score": -1, "input_data_size": 16342041}
>2018-07-11 12:23:41.364850 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 987] Generated table #10608: 3330 keys, 16186500 bytes
>2018-07-11 12:23:41.364891 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821364865, "cf_name": "default", "job": 987, "event": "table_file_creation", "file_number": 10608, "file_size": 16186500, "table_properties": {"data_size": 16063489, "index_size": 70083, "filter_size": 52001, "raw_key_size": 66928, "raw_average_key_size": 20, "raw_value_size": 15967106, "raw_average_value_size": 4794, "num_data_blocks": 2058, "num_entries": 3330, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:23:41.406373 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 987] Compacted 1@0 + 1@1 files to L1 => 16186500 bytes
>2018-07-11 12:23:41.439897 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.439831) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 70.2 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(1.8, 13.7) out(15.4), read-write-amplify(16.9) write-amplify(8.4) OK, records in: 3612, records dropped: 282
>
>2018-07-11 12:23:41.439907 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:41.439852) EVENT_LOG_v1 {"time_micros": 1531311821439842, "job": 987, "event": "compaction_finished", "compaction_time_micros": 232760, "output_level": 1, "num_output_files": 1, "total_output_size": 16186500, "num_input_records": 3612, "num_output_records": 3330, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:23:41.440366 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821440362, "job": 987, "event": "table_file_deletion", "file_number": 10607}
>2018-07-11 12:23:41.443204 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311821443202, "job": 987, "event": "table_file_deletion", "file_number": 10605}
>2018-07-11 12:23:41.443247 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:23:56.101423 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10609. Immutable memtables: 0.
>
>2018-07-11 12:23:56.101463 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 988] Syncing log #10606
>2018-07-11 12:23:56.117898 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.101454) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:23:56.117910 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 988] Flushing memtable with next log file: 10609
>
>2018-07-11 12:23:56.117924 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836117916, "job": 988, "event": "flush_started", "num_memtables": 1, "num_entries": 312, "num_deletes": 251, "memory_usage": 139568}
>2018-07-11 12:23:56.117927 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 988] Level-0 flush table #10610: started
>2018-07-11 12:23:56.160152 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836160139, "cf_name": "default", "job": 988, "event": "table_file_creation", "file_number": 10610, "file_size": 128849, "table_properties": {"data_size": 126312, "index_size": 576, "filter_size": 1042, "raw_key_size": 5532, "raw_average_key_size": 20, "raw_value_size": 122238, "raw_average_value_size": 446, "num_data_blocks": 18, "num_entries": 274, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:23:56.160162 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 988] Level-0 flush table #10610: 128849 bytes OK
>2018-07-11 12:23:56.218460 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.176713) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10610 started
>2018-07-11 12:23:56.218469 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.218425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10610: memtable #1 done
>2018-07-11 12:23:56.218472 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.218440) EVENT_LOG_v1 {"time_micros": 1531311836218434, "job": 988, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:23:56.218474 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.218451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:23:56.218479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 988] Try to delete WAL files size 131129, prev total WAL file size 131129, number of live WAL files 2.
>
>2018-07-11 12:23:56.218595 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:23:56.218643 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.218636) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:23:56.218652 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 989] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:23:56.218658 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 988 Base level 0, inputs: [10610(125KB)], [10608(15MB)]
>
>2018-07-11 12:23:56.218672 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836218662, "job": 989, "event": "compaction_started", "files_L0": [10610], "files_L1": [10608], "score": -1, "input_data_size": 16315349}
>2018-07-11 12:23:56.394256 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 989] Generated table #10611: 3096 keys, 14314531 bytes
>2018-07-11 12:23:56.394298 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836394283, "cf_name": "default", "job": 989, "event": "table_file_creation", "file_number": 10611, "file_size": 14314531, "table_properties": {"data_size": 14204939, "index_size": 62402, "filter_size": 46263, "raw_key_size": 62264, "raw_average_key_size": 20, "raw_value_size": 14117199, "raw_average_value_size": 4559, "num_data_blocks": 1824, "num_entries": 3096, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:23:56.435764 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 989] Compacted 1@0 + 1@1 files to L1 => 14314531 bytes
>2018-07-11 12:23:56.477634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.477563) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.2 rd, 65.9 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.4) out(13.7), read-write-amplify(237.7) write-amplify(111.1) OK, records in: 3604, records dropped: 508
>
>2018-07-11 12:23:56.477643 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:23:56.477585) EVENT_LOG_v1 {"time_micros": 1531311836477575, "job": 989, "event": "compaction_finished", "compaction_time_micros": 217076, "output_level": 1, "num_output_files": 1, "total_output_size": 14314531, "num_input_records": 3604, "num_output_records": 3096, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:23:56.477756 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836477751, "job": 989, "event": "table_file_deletion", "file_number": 10610}
>2018-07-11 12:23:56.480727 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531311836480724, "job": 989, "event": "table_file_deletion", "file_number": 10608}
>2018-07-11 12:24:03.649573 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:25:03.649751 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:26:03.649922 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:27:03.650093 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:28:03.650275 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:29:03.650445 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:30:03.650622 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:31:03.650801 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:32:01.269619 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10612. Immutable memtables: 0.
>
>2018-07-11 12:32:01.269661 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 990] Syncing log #10609
>2018-07-11 12:32:01.285978 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.269651) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:32:01.285989 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 990] Flushing memtable with next log file: 10612
>
>2018-07-11 12:32:01.286015 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321286007, "job": 990, "event": "flush_started", "num_memtables": 1, "num_entries": 1949, "num_deletes": 250, "memory_usage": 3644656}
>2018-07-11 12:32:01.286018 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 990] Level-0 flush table #10613: started
>2018-07-11 12:32:01.336510 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321336495, "cf_name": "default", "job": 990, "event": "table_file_creation", "file_number": 10613, "file_size": 1869951, "table_properties": {"data_size": 1854523, "index_size": 8000, "filter_size": 6508, "raw_key_size": 10650, "raw_average_key_size": 21, "raw_value_size": 1841826, "raw_average_value_size": 3647, "num_data_blocks": 251, "num_entries": 505, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:32:01.336521 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 990] Level-0 flush table #10613: 1869951 bytes OK
>2018-07-11 12:32:01.386557 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.353047) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10613 started
>2018-07-11 12:32:01.386565 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.386520) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10613: memtable #1 done
>2018-07-11 12:32:01.386568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.386535) EVENT_LOG_v1 {"time_micros": 1531312321386528, "job": 990, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:32:01.386582 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.386547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:32:01.386590 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 990] Try to delete WAL files size 3607205, prev total WAL file size 3607205, number of live WAL files 2.
>
>2018-07-11 12:32:01.387479 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:32:01.387540 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.387533) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:32:01.387548 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 991] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:32:01.387554 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 990 Base level 0, inputs: [10613(1826KB)], [10611(13MB)]
>
>2018-07-11 12:32:01.387567 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321387557, "job": 991, "event": "compaction_started", "files_L0": [10613], "files_L1": [10611], "score": -1, "input_data_size": 16184482}
>2018-07-11 12:32:01.580155 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 991] Generated table #10614: 3331 keys, 16083607 bytes
>2018-07-11 12:32:01.580216 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321580199, "cf_name": "default", "job": 991, "event": "table_file_creation", "file_number": 10614, "file_size": 16083607, "table_properties": {"data_size": 15960757, "index_size": 70113, "filter_size": 51810, "raw_key_size": 66948, "raw_average_key_size": 20, "raw_value_size": 15864338, "raw_average_value_size": 4762, "num_data_blocks": 2059, "num_entries": 3331, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:32:01.613316 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 991] Compacted 1@0 + 1@1 files to L1 => 16083607 bytes
>2018-07-11 12:32:01.655175 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.655106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.7 rd, 71.2 wr, level 1, files in(1, 1) out(1) MB in(1.8, 13.7) out(15.3), read-write-amplify(17.3) write-amplify(8.6) OK, records in: 3601, records dropped: 270
>
>2018-07-11 12:32:01.655195 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:01.655127) EVENT_LOG_v1 {"time_micros": 1531312321655117, "job": 991, "event": "compaction_finished", "compaction_time_micros": 225735, "output_level": 1, "num_output_files": 1, "total_output_size": 16083607, "num_input_records": 3601, "num_output_records": 3331, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:32:01.655620 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321655616, "job": 991, "event": "table_file_deletion", "file_number": 10613}
>2018-07-11 12:32:01.658452 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312321658450, "job": 991, "event": "table_file_deletion", "file_number": 10611}
>2018-07-11 12:32:01.658515 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:32:03.650990 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 12:32:14.306959 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10615. Immutable memtables: 0.
>
>2018-07-11 12:32:14.307000 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 992] Syncing log #10612
>2018-07-11 12:32:14.323546 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.306991) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:32:14.323557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 992] Flushing memtable with next log file: 10615
>
>2018-07-11 12:32:14.323584 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334323564, "job": 992, "event": "flush_started", "num_memtables": 1, "num_entries": 305, "num_deletes": 251, "memory_usage": 124576}
>2018-07-11 12:32:14.323588 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 992] Level-0 flush table #10616: started
>2018-07-11 12:32:14.357409 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334357396, "cf_name": "default", "job": 992, "event": "table_file_creation", "file_number": 10616, "file_size": 114557, "table_properties": {"data_size": 112132, "index_size": 510, "filter_size": 996, "raw_key_size": 5490, "raw_average_key_size": 20, "raw_value_size": 108137, "raw_average_value_size": 397, "num_data_blocks": 16, "num_entries": 272, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:32:14.357419 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 992] Level-0 flush table #10616: 114557 bytes OK
>2018-07-11 12:32:14.407459 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.373958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10616 started
>2018-07-11 12:32:14.407467 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.407424) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10616: memtable #1 done
>2018-07-11 12:32:14.407469 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.407439) EVENT_LOG_v1 {"time_micros": 1531312334407433, "job": 992, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:32:14.407483 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.407450) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:32:14.407489 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 992] Try to delete WAL files size 116304, prev total WAL file size 116304, number of live WAL files 2.
>
>2018-07-11 12:32:14.407602 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:32:14.407650 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.407644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:32:14.407659 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 993] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:32:14.407666 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 992 Base level 0, inputs: [10616(111KB)], [10614(15MB)]
>
>2018-07-11 12:32:14.407679 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334407670, "job": 993, "event": "compaction_started", "files_L0": [10616], "files_L1": [10614], "score": -1, "input_data_size": 16198164}
>2018-07-11 12:32:14.574828 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 993] Generated table #10617: 3095 keys, 14226434 bytes
>2018-07-11 12:32:14.574872 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334574846, "cf_name": "default", "job": 993, "event": "table_file_creation", "file_number": 10617, "file_size": 14226434, "table_properties": {"data_size": 14117058, "index_size": 62367, "filter_size": 46082, "raw_key_size": 62242, "raw_average_key_size": 20, "raw_value_size": 14029357, "raw_average_value_size": 4532, "num_data_blocks": 1823, "num_entries": 3095, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:32:14.616343 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 993] Compacted 1@0 + 1@1 files to L1 => 14226434 bytes
>2018-07-11 12:32:14.649875 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.649806) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.6 rd, 68.2 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.3) out(13.6), read-write-amplify(265.6) write-amplify(124.2) OK, records in: 3603, records dropped: 508
>
>2018-07-11 12:32:14.649883 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:32:14.649828) EVENT_LOG_v1 {"time_micros": 1531312334649818, "job": 993, "event": "compaction_finished", "compaction_time_micros": 208648, "output_level": 1, "num_output_files": 1, "total_output_size": 14226434, "num_input_records": 3603, "num_output_records": 3095, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:32:14.649997 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334649992, "job": 993, "event": "table_file_deletion", "file_number": 10616}
>2018-07-11 12:32:14.652898 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312334652895, "job": 993, "event": "table_file_deletion", "file_number": 10614}
>2018-07-11 12:32:19.342374 7f29149e4700  1 mon.magna102@0(leader).log v21303 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21303)
>2018-07-11 12:32:29.318244 7f29149e4700  1 mon.magna102@0(leader).log v21304 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21304)
>2018-07-11 12:33:03.651170 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:34:03.651354 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:35:03.651513 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:36:03.651694 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:37:03.651874 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:38:03.652011 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:39:03.652204 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:40:03.652376 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:40:21.493400 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10618. Immutable memtables: 0.
>
>2018-07-11 12:40:21.493444 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 994] Syncing log #10615
>2018-07-11 12:40:21.509825 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.493435) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:40:21.509836 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 994] Flushing memtable with next log file: 10618
>
>2018-07-11 12:40:21.509862 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821509854, "job": 994, "event": "flush_started", "num_memtables": 1, "num_entries": 1972, "num_deletes": 250, "memory_usage": 3699952}
>2018-07-11 12:40:21.509865 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 994] Level-0 flush table #10619: started
>2018-07-11 12:40:21.568692 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821568678, "cf_name": "default", "job": 994, "event": "table_file_creation", "file_number": 10619, "file_size": 1916065, "table_properties": {"data_size": 1900375, "index_size": 8131, "filter_size": 6639, "raw_key_size": 10843, "raw_average_key_size": 21, "raw_value_size": 1887439, "raw_average_value_size": 3672, "num_data_blocks": 255, "num_entries": 514, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:40:21.568702 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 994] Level-0 flush table #10619: 1916065 bytes OK
>2018-07-11 12:40:21.618714 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.585234) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10619 started
>2018-07-11 12:40:21.618722 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.618678) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10619: memtable #1 done
>2018-07-11 12:40:21.618725 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.618693) EVENT_LOG_v1 {"time_micros": 1531312821618687, "job": 994, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:40:21.618738 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.618704) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:40:21.618745 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 994] Try to delete WAL files size 3661938, prev total WAL file size 3661938, number of live WAL files 2.
>
>2018-07-11 12:40:21.619704 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:40:21.619896 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 12:40:21.619903 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 74848.5 total, 1000.4 interval
>Cumulative writes: 99K writes, 471K keys, 99K commit groups, 1.0 writes per commit group, ingest: 0.78 GB, 0.01 MB/s
>Cumulative WAL: 99K writes, 99K syncs, 1.00 writes per sync, written: 0.78 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1012 writes, 4538 keys, 1012 commit groups, 1.0 writes per commit group, ingest: 7.17 MB, 0.01 MB/s
>Interval WAL: 1012 writes, 1012 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.83 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       498    0.080       0      0
>  L1      1/1   13.57 MB   0.0      7.8     0.7      7.2       7.2     -0.0       0.0  10.7     72.7     66.5       111       495    0.223   1779K   213K
> Sum      2/2   15.39 MB   0.0      7.8     0.7      7.2       7.8      0.7       0.0  11.7     53.4     53.4       150       993    0.152   1779K   213K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.1     54.6     54.4         1         8    0.142     14K   1568
>Uptime(secs): 74848.5 total, 74848.5 interval
>Flush(GB): cumulative 0.670, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.85 GB write, 0.11 MB/s write, 7.85 GB read, 0.11 MB/s read, 150.5 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.83 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       498    0.080       0      0
>  L1      1/1   13.57 MB   0.0      7.8     0.7      7.2       7.2     -0.0       0.0  10.7     72.7     66.5       111       495    0.223   1779K   213K
> Sum      2/2   15.39 MB   0.0      7.8     0.7      7.2       7.8      0.7       0.0  11.7     53.4     53.4       150       993    0.152   1779K   213K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 74848.5 total, 74848.5 interval
>Flush(GB): cumulative 0.670, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.85 GB write, 0.11 MB/s write, 7.85 GB read, 0.11 MB/s read, 150.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 12:40:21.619924 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.619918) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:40:21.619929 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 995] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:40:21.619934 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 994 Base level 0, inputs: [10619(1871KB)], [10617(13MB)]
>
>2018-07-11 12:40:21.619946 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821619937, "job": 995, "event": "compaction_started", "files_L0": [10619], "files_L1": [10617], "score": -1, "input_data_size": 16142499}
>2018-07-11 12:40:21.811074 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 995] Generated table #10620: 3338 keys, 16048435 bytes
>2018-07-11 12:40:21.811124 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821811096, "cf_name": "default", "job": 995, "event": "table_file_creation", "file_number": 10620, "file_size": 16048435, "table_properties": {"data_size": 15925435, "index_size": 70285, "filter_size": 51788, "raw_key_size": 67090, "raw_average_key_size": 20, "raw_value_size": 15828787, "raw_average_value_size": 4741, "num_data_blocks": 2064, "num_entries": 3338, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:40:21.844266 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 995] Compacted 1@0 + 1@1 files to L1 => 16048435 bytes
>2018-07-11 12:40:21.877807 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.877739) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 71.5 wr, level 1, files in(1, 1) out(1) MB in(1.8, 13.6) out(15.3), read-write-amplify(16.8) write-amplify(8.4) OK, records in: 3609, records dropped: 271
>
>2018-07-11 12:40:21.877816 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:21.877760) EVENT_LOG_v1 {"time_micros": 1531312821877751, "job": 995, "event": "compaction_finished", "compaction_time_micros": 224301, "output_level": 1, "num_output_files": 1, "total_output_size": 16048435, "num_input_records": 3609, "num_output_records": 3338, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:40:21.878265 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821878260, "job": 995, "event": "table_file_deletion", "file_number": 10619}
>2018-07-11 12:40:21.881094 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312821881091, "job": 995, "event": "table_file_deletion", "file_number": 10617}
>2018-07-11 12:40:21.881214 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:40:28.520873 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10621. Immutable memtables: 0.
>
>2018-07-11 12:40:28.520936 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 996] Syncing log #10618
>2018-07-11 12:40:28.537555 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.520926) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:40:28.537586 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 996] Flushing memtable with next log file: 10621
>
>2018-07-11 12:40:28.537600 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828537593, "job": 996, "event": "flush_started", "num_memtables": 1, "num_entries": 284, "num_deletes": 251, "memory_usage": 79816}
>2018-07-11 12:40:28.537603 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 996] Level-0 flush table #10622: started
>2018-07-11 12:40:28.571376 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828571363, "cf_name": "default", "job": 996, "event": "table_file_creation", "file_number": 10622, "file_size": 71636, "table_properties": {"data_size": 69548, "index_size": 311, "filter_size": 858, "raw_key_size": 5364, "raw_average_key_size": 20, "raw_value_size": 65778, "raw_average_value_size": 247, "num_data_blocks": 10, "num_entries": 266, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:40:28.571387 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 996] Level-0 flush table #10622: 71636 bytes OK
>2018-07-11 12:40:28.621378 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.587882) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10622 started
>2018-07-11 12:40:28.621386 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.621345) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10622: memtable #1 done
>2018-07-11 12:40:28.621389 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.621359) EVENT_LOG_v1 {"time_micros": 1531312828621353, "job": 996, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:40:28.621404 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.621370) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:40:28.621409 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 996] Try to delete WAL files size 71773, prev total WAL file size 71773, number of live WAL files 2.
>
>2018-07-11 12:40:28.621515 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:40:28.621561 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.621555) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:40:28.621570 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 997] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:40:28.621576 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 996 Base level 0, inputs: [10622(69KB)], [10620(15MB)]
>
>2018-07-11 12:40:28.621590 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828621580, "job": 997, "event": "compaction_started", "files_L0": [10622], "files_L1": [10620], "score": -1, "input_data_size": 16120071}
>2018-07-11 12:40:28.797105 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 997] Generated table #10623: 3096 keys, 14221158 bytes
>2018-07-11 12:40:28.797149 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828797122, "cf_name": "default", "job": 997, "event": "table_file_creation", "file_number": 10623, "file_size": 14221158, "table_properties": {"data_size": 14111785, "index_size": 62367, "filter_size": 46079, "raw_key_size": 62258, "raw_average_key_size": 20, "raw_value_size": 14024059, "raw_average_value_size": 4529, "num_data_blocks": 1823, "num_entries": 3096, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:40:28.830282 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 997] Compacted 1@0 + 1@1 files to L1 => 14221158 bytes
>2018-07-11 12:40:28.863816 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.863748) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.2 rd, 68.1 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.3) out(13.6), read-write-amplify(423.5) write-amplify(198.5) OK, records in: 3604, records dropped: 508
>
>2018-07-11 12:40:28.863824 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:40:28.863770) EVENT_LOG_v1 {"time_micros": 1531312828863760, "job": 997, "event": "compaction_finished", "compaction_time_micros": 208676, "output_level": 1, "num_output_files": 1, "total_output_size": 14221158, "num_input_records": 3604, "num_output_records": 3096, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:40:28.863928 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828863924, "job": 997, "event": "table_file_deletion", "file_number": 10622}
>2018-07-11 12:40:28.866854 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531312828866851, "job": 997, "event": "table_file_deletion", "file_number": 10620}
>2018-07-11 12:41:03.652559 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:42:03.652736 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:43:03.652919 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:44:03.653100 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:45:03.653284 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:46:03.653461 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:46:07.668843 7f29149e4700  1 mon.magna102@0(leader).log v21305 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21305)
>2018-07-11 12:46:09.802476 7f29149e4700  1 mon.magna102@0(leader).log v21306 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21306)
>2018-07-11 12:46:17.903173 7f29149e4700  1 mon.magna102@0(leader).log v21307 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21307)
>2018-07-11 12:46:30.546490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:46:30.696081 7f29149e4700  1 mon.magna102@0(leader).log v21308 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21308)
>2018-07-11 12:46:49.285612 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:46:49.285636 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3066121695' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:46:49.707049 7f29149e4700  1 mon.magna102@0(leader).log v21309 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21309)
>2018-07-11 12:46:50.903146 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:46:50.903172 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3370778953' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:46:50.904524 7f29191ed700  1 mon.magna102@0(leader).log v21309 check_sub sending message to client.? 10.8.128.102:0/3370778953 with 1 entries (version 21309)
>2018-07-11 12:46:51.874053 7f29149e4700  1 mon.magna102@0(leader).log v21310 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21310)
>2018-07-11 12:46:51.874086 7f29149e4700  1 mon.magna102@0(leader).log v21310 check_sub sending message to client.? 10.8.128.102:0/3370778953 with 0 entries (version 21310)
>2018-07-11 12:47:03.653643 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:47:14.892866 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:47:14.892888 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3820135455' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:47:15.701479 7f29149e4700  1 mon.magna102@0(leader).log v21311 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21311)
>2018-07-11 12:47:20.228622 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 12:47:20.228650 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4197207835' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 12:47:20.343615 7f29149e4700  1 mon.magna102@0(leader).log v21312 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21312)
>2018-07-11 12:47:27.935092 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd tree"} v 0) v1
>2018-07-11 12:47:27.935118 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3594135888' entity='client.admin' cmd=[{"prefix": "osd tree"}]: dispatch
>2018-07-11 12:47:28.044622 7f29149e4700  1 mon.magna102@0(leader).log v21313 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21313)
>2018-07-11 12:47:44.580173 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 12:47:44.730714 7f29149e4700  1 mon.magna102@0(leader).log v21314 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21314)
>2018-07-11 12:47:50.582702 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 10)
>2018-07-11 12:47:50.739754 7f29149e4700  1 mon.magna102@0(leader).log v21315 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21315)
>2018-07-11 12:48:03.653825 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:48:04.588687 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:48:04.740861 7f29149e4700  1 mon.magna102@0(leader).log v21316 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21316)
>2018-07-11 12:48:24.742755 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10624. Immutable memtables: 0.
>
>2018-07-11 12:48:24.742799 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 998] Syncing log #10621
>2018-07-11 12:48:24.759335 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.742790) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:48:24.759346 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 998] Flushing memtable with next log file: 10624
>
>2018-07-11 12:48:24.759374 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313304759365, "job": 998, "event": "flush_started", "num_memtables": 1, "num_entries": 2018, "num_deletes": 251, "memory_usage": 3825616}
>2018-07-11 12:48:24.759377 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 998] Level-0 flush table #10625: started
>2018-07-11 12:48:24.826586 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313304826559, "cf_name": "default", "job": 998, "event": "table_file_creation", "file_number": 10625, "file_size": 3652559, "table_properties": {"data_size": 3622679, "index_size": 16702, "filter_size": 12253, "raw_key_size": 15975, "raw_average_key_size": 20, "raw_value_size": 3599993, "raw_average_value_size": 4663, "num_data_blocks": 502, "num_entries": 772, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:48:24.826597 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 998] Level-0 flush table #10625: 3652559 bytes OK
>2018-07-11 12:48:24.876553 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.843055) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10625 started
>2018-07-11 12:48:24.876561 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.876514) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10625: memtable #1 done
>2018-07-11 12:48:24.876564 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.876528) EVENT_LOG_v1 {"time_micros": 1531313304876522, "job": 998, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:48:24.876579 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.876542) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:48:24.876586 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 998] Try to delete WAL files size 3786589, prev total WAL file size 3786589, number of live WAL files 2.
>
>2018-07-11 12:48:24.877510 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:48:24.877574 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:24.877567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:48:24.877583 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 999] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:48:24.877589 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 998 Base level 0, inputs: [10625(3566KB)], [10623(13MB)]
>
>2018-07-11 12:48:24.877602 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313304877593, "job": 999, "event": "compaction_started", "files_L0": [10625], "files_L1": [10623], "score": -1, "input_data_size": 17873717}
>2018-07-11 12:48:25.068915 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 999] Generated table #10626: 3358 keys, 15983561 bytes
>2018-07-11 12:48:25.068963 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313305068935, "cf_name": "default", "job": 999, "event": "table_file_creation", "file_number": 10626, "file_size": 15983561, "table_properties": {"data_size": 15859730, "index_size": 71132, "filter_size": 51772, "raw_key_size": 67986, "raw_average_key_size": 20, "raw_value_size": 15762066, "raw_average_value_size": 4693, "num_data_blocks": 2074, "num_entries": 3358, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:48:25.102069 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 999] Compacted 1@0 + 1@1 files to L1 => 15983561 bytes
>2018-07-11 12:48:25.135658 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:25.135583) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.6 rd, 71.2 wr, level 1, files in(1, 1) out(1) MB in(3.5, 13.6) out(15.2), read-write-amplify(9.3) write-amplify(4.4) OK, records in: 3868, records dropped: 510
>
>2018-07-11 12:48:25.135667 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:25.135605) EVENT_LOG_v1 {"time_micros": 1531313305135595, "job": 999, "event": "compaction_finished", "compaction_time_micros": 224449, "output_level": 1, "num_output_files": 1, "total_output_size": 15983561, "num_input_records": 3868, "num_output_records": 3358, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:48:25.136486 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313305136482, "job": 999, "event": "table_file_deletion", "file_number": 10625}
>2018-07-11 12:48:25.139289 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313305139286, "job": 999, "event": "table_file_deletion", "file_number": 10623}
>2018-07-11 12:48:41.721965 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10627. Immutable memtables: 0.
>
>2018-07-11 12:48:41.722027 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1000] Syncing log #10624
>2018-07-11 12:48:41.738718 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.722017) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:48:41.738729 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1000] Flushing memtable with next log file: 10627
>
>2018-07-11 12:48:41.738755 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313321738735, "job": 1000, "event": "flush_started", "num_memtables": 1, "num_entries": 311, "num_deletes": 250, "memory_usage": 143376}
>2018-07-11 12:48:41.738758 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1000] Level-0 flush table #10628: started
>2018-07-11 12:48:41.780925 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313321780913, "cf_name": "default", "job": 1000, "event": "table_file_creation", "file_number": 10628, "file_size": 117845, "table_properties": {"data_size": 115381, "index_size": 539, "filter_size": 1006, "raw_key_size": 5970, "raw_average_key_size": 22, "raw_value_size": 111343, "raw_average_value_size": 410, "num_data_blocks": 17, "num_entries": 271, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:48:41.780948 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1000] Level-0 flush table #10628: 117845 bytes OK
>2018-07-11 12:48:41.830977 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.797473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10628 started
>2018-07-11 12:48:41.830985 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.830943) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10628: memtable #1 done
>2018-07-11 12:48:41.830987 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.830957) EVENT_LOG_v1 {"time_micros": 1531313321830951, "job": 1000, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:48:41.830989 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.830968) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:48:41.831007 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1000] Try to delete WAL files size 135435, prev total WAL file size 135435, number of live WAL files 2.
>
>2018-07-11 12:48:41.831123 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:48:41.831169 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:41.831162) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:48:41.831188 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1001] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:48:41.831194 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1000 Base level 0, inputs: [10628(115KB)], [10626(15MB)]
>
>2018-07-11 12:48:41.831207 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313321831198, "job": 1001, "event": "compaction_started", "files_L0": [10628], "files_L1": [10626], "score": -1, "input_data_size": 16101406}
>2018-07-11 12:48:42.006655 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1001] Generated table #10629: 3125 keys, 14401481 bytes
>2018-07-11 12:48:42.006703 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313322006675, "cf_name": "default", "job": 1001, "event": "table_file_creation", "file_number": 10629, "file_size": 14401481, "table_properties": {"data_size": 14290958, "index_size": 62985, "filter_size": 46611, "raw_key_size": 62842, "raw_average_key_size": 20, "raw_value_size": 14202399, "raw_average_value_size": 4544, "num_data_blocks": 1841, "num_entries": 3125, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:48:42.145039 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1001] Compacted 1@0 + 1@1 files to L1 => 14401481 bytes
>2018-07-11 12:48:42.173412 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:42.173334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 51.3 rd, 45.9 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.2) out(13.7), read-write-amplify(258.8) write-amplify(122.2) OK, records in: 3629, records dropped: 504
>
>2018-07-11 12:48:42.173421 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:48:42.173356) EVENT_LOG_v1 {"time_micros": 1531313322173346, "job": 1001, "event": "compaction_finished", "compaction_time_micros": 313817, "output_level": 1, "num_output_files": 1, "total_output_size": 14401481, "num_input_records": 3629, "num_output_records": 3125, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:48:42.173532 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313322173527, "job": 1001, "event": "table_file_deletion", "file_number": 10628}
>2018-07-11 12:48:42.176442 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313322176439, "job": 1001, "event": "table_file_deletion", "file_number": 10626}
>2018-07-11 12:48:42.176559 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:49:03.654001 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:49:14.619917 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:49:14.793161 7f29149e4700  1 mon.magna102@0(leader).log v21317 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21317)
>2018-07-11 12:49:24.624288 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:49:24.794122 7f29149e4700  1 mon.magna102@0(leader).log v21318 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21318)
>2018-07-11 12:49:44.632915 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:49:44.780152 7f29149e4700  1 mon.magna102@0(leader).log v21319 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21319)
>2018-07-11 12:50:03.654187 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:50:54.663530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:50:54.820084 7f29149e4700  1 mon.magna102@0(leader).log v21320 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21320)
>2018-07-11 12:51:03.654374 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:51:04.667913 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:51:04.837475 7f29149e4700  1 mon.magna102@0(leader).log v21321 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21321)
>2018-07-11 12:51:20.674739 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:51:20.847022 7f29149e4700  1 mon.magna102@0(leader).log v21322 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21322)
>2018-07-11 12:52:03.654548 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:52:34.707228 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:52:34.861634 7f29149e4700  1 mon.magna102@0(leader).log v21323 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21323)
>2018-07-11 12:52:44.711486 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:52:44.862444 7f29149e4700  1 mon.magna102@0(leader).log v21324 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21324)
>2018-07-11 12:53:00.718475 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:53:00.889909 7f29149e4700  1 mon.magna102@0(leader).log v21325 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21325)
>2018-07-11 12:53:03.654722 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:54:03.654897 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:54:10.749166 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:54:10.903008 7f29149e4700  1 mon.magna102@0(leader).log v21326 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21326)
>2018-07-11 12:54:24.755190 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:54:24.913670 7f29149e4700  1 mon.magna102@0(leader).log v21327 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21327)
>2018-07-11 12:54:34.759509 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:54:34.914490 7f29149e4700  1 mon.magna102@0(leader).log v21328 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21328)
>2018-07-11 12:55:03.655075 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:55:44.790102 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:55:44.937276 7f29149e4700  1 mon.magna102@0(leader).log v21329 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21329)
>2018-07-11 12:56:00.796963 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:56:00.946694 7f29149e4700  1 mon.magna102@0(leader).log v21330 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21330)
>2018-07-11 12:56:03.655255 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 12:56:14.802974 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:56:14.931436 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10630. Immutable memtables: 0.
>
>2018-07-11 12:56:14.964723 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:56:14.931494) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:56:14.964735 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1002] Flushing memtable with next log file: 10630
>
>2018-07-11 12:56:14.964750 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313774964742, "job": 1002, "event": "flush_started", "num_memtables": 1, "num_entries": 1957, "num_deletes": 251, "memory_usage": 3754392}
>2018-07-11 12:56:14.964753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1002] Level-0 flush table #10631: started
>2018-07-11 12:56:15.042387 7f29149e4700  1 mon.magna102@0(leader).log v21331 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21331)
>2018-07-11 12:56:15.065384 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313775065369, "cf_name": "default", "job": 1002, "event": "table_file_creation", "file_number": 10631, "file_size": 3563973, "table_properties": {"data_size": 3535021, "index_size": 16100, "filter_size": 11928, "raw_key_size": 15635, "raw_average_key_size": 20, "raw_value_size": 3512986, "raw_average_value_size": 4646, "num_data_blocks": 484, "num_entries": 756, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 12:56:15.065395 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1002] Level-0 flush table #10631: 3563973 bytes OK
>2018-07-11 12:56:15.115389 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.081909) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10631 started
>2018-07-11 12:56:15.115397 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.115349) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10631: memtable #1 done
>2018-07-11 12:56:15.115400 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.115364) EVENT_LOG_v1 {"time_micros": 1531313775115358, "job": 1002, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:56:15.115402 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.115378) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:56:15.115409 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1002] Try to delete WAL files size 3716000, prev total WAL file size 3733926, number of live WAL files 2.
>
>2018-07-11 12:56:15.116266 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:56:15.116453 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 12:56:15.116460 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 75802.0 total, 953.5 interval
>Cumulative writes: 100K writes, 476K keys, 100K commit groups, 1.0 writes per commit group, ingest: 0.79 GB, 0.01 MB/s
>Cumulative WAL: 100K writes, 100K syncs, 1.00 writes per sync, written: 0.79 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1016 writes, 4578 keys, 1016 commit groups, 1.0 writes per commit group, ingest: 7.37 MB, 0.01 MB/s
>Interval WAL: 1016 writes, 1016 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.40 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       502    0.080       0      0
>  L1      1/1   13.73 MB   0.0      7.9     0.7      7.2       7.2     -0.0       0.0  10.7     72.7     66.4       111       499    0.223   1794K   215K
> Sum      2/2   17.13 MB   0.0      7.9     0.7      7.2       7.9      0.7       0.0  11.7     53.4     53.4       152      1001    0.152   1794K   215K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.2     49.3     50.7         1         8    0.160     14K   1793
>Uptime(secs): 75802.0 total, 75802.0 interval
>Flush(GB): cumulative 0.677, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.91 GB write, 0.11 MB/s write, 7.91 GB read, 0.11 MB/s read, 151.8 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.3 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.40 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        40       502    0.080       0      0
>  L1      1/1   13.73 MB   0.0      7.9     0.7      7.2       7.2     -0.0       0.0  10.7     72.7     66.4       111       499    0.223   1794K   215K
> Sum      2/2   17.13 MB   0.0      7.9     0.7      7.2       7.9      0.7       0.0  11.7     53.4     53.4       152      1001    0.152   1794K   215K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 75802.0 total, 75802.0 interval
>Flush(GB): cumulative 0.677, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.91 GB write, 0.11 MB/s write, 7.91 GB read, 0.11 MB/s read, 151.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 12:56:15.116481 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.116474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 12:56:15.116486 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1003] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:56:15.116492 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1002 Base level 0, inputs: [10631(3480KB)], [10629(13MB)]
>
>2018-07-11 12:56:15.116504 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313775116495, "job": 1003, "event": "compaction_started", "files_L0": [10631], "files_L1": [10629], "score": -1, "input_data_size": 17965454}
>2018-07-11 12:56:15.307654 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1003] Generated table #10632: 3371 keys, 16070010 bytes
>2018-07-11 12:56:15.307701 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313775307673, "cf_name": "default", "job": 1003, "event": "table_file_creation", "file_number": 10632, "file_size": 16070010, "table_properties": {"data_size": 15945966, "index_size": 71144, "filter_size": 51973, "raw_key_size": 68230, "raw_average_key_size": 20, "raw_value_size": 15848107, "raw_average_value_size": 4701, "num_data_blocks": 2074, "num_entries": 3371, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:56:15.340979 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1003] Compacted 1@0 + 1@1 files to L1 => 16070010 bytes
>2018-07-11 12:56:15.382834 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.382767) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.0 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(3.4, 13.7) out(15.3), read-write-amplify(9.5) write-amplify(4.5) OK, records in: 3881, records dropped: 510
>
>2018-07-11 12:56:15.382843 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:56:15.382788) EVENT_LOG_v1 {"time_micros": 1531313775382778, "job": 1003, "event": "compaction_finished", "compaction_time_micros": 224458, "output_level": 1, "num_output_files": 1, "total_output_size": 16070010, "num_input_records": 3881, "num_output_records": 3371, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:56:15.383583 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313775383579, "job": 1003, "event": "table_file_deletion", "file_number": 10631}
>2018-07-11 12:56:15.386425 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313775386422, "job": 1003, "event": "table_file_deletion", "file_number": 10629}
>2018-07-11 12:56:39.128359 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:56:39.128384 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4240430199' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:56:39.129758 7f29191ed700  1 mon.magna102@0(leader).log v21331 check_sub sending message to client.? 10.8.128.102:0/4240430199 with 1 entries (version 21331)
>2018-07-11 12:56:39.944602 7f29149e4700  1 mon.magna102@0(leader).log v21332 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21332)
>2018-07-11 12:56:39.944635 7f29149e4700  1 mon.magna102@0(leader).log v21332 check_sub sending message to client.? 10.8.128.102:0/4240430199 with 0 entries (version 21332)
>2018-07-11 12:57:01.921669 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10633. Immutable memtables: 0.
>
>2018-07-11 12:57:01.921713 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1004] Syncing log #10630
>2018-07-11 12:57:01.938216 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:57:01.921703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 12:57:01.938228 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1004] Flushing memtable with next log file: 10633
>
>2018-07-11 12:57:01.938254 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313821938246, "job": 1004, "event": "flush_started", "num_memtables": 1, "num_entries": 432, "num_deletes": 250, "memory_usage": 410216}
>2018-07-11 12:57:01.938257 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1004] Level-0 flush table #10634: started
>2018-07-11 12:57:01.972101 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313821972086, "cf_name": "default", "job": 1004, "event": "table_file_creation", "file_number": 10634, "file_size": 269375, "table_properties": {"data_size": 265875, "index_size": 1114, "filter_size": 1467, "raw_key_size": 6443, "raw_average_key_size": 21, "raw_value_size": 261077, "raw_average_value_size": 888, "num_data_blocks": 35, "num_entries": 294, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 12:57:01.972123 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1004] Level-0 flush table #10634: 269375 bytes OK
>2018-07-11 12:57:02.022116 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:57:01.988622) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10634 started
>2018-07-11 12:57:02.022124 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.022080) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10634: memtable #1 done
>2018-07-11 12:57:02.022139 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.022095) EVENT_LOG_v1 {"time_micros": 1531313822022089, "job": 1004, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 12:57:02.022142 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.022106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 12:57:02.022148 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1004] Try to delete WAL files size 400348, prev total WAL file size 400348, number of live WAL files 2.
>
>2018-07-11 12:57:02.022335 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:57:02.022384 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.022377) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 12:57:02.022394 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1005] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 12:57:02.022400 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1004 Base level 0, inputs: [10634(263KB)], [10632(15MB)]
>
>2018-07-11 12:57:02.022413 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313822022403, "job": 1005, "event": "compaction_started", "files_L0": [10634], "files_L1": [10632], "score": -1, "input_data_size": 16339385}
>2018-07-11 12:57:02.206211 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1005] Generated table #10635: 3174 keys, 14736738 bytes
>2018-07-11 12:57:02.206261 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313822206232, "cf_name": "default", "job": 1005, "event": "table_file_creation", "file_number": 10635, "file_size": 14736738, "table_properties": {"data_size": 14624060, "index_size": 64150, "filter_size": 47601, "raw_key_size": 63838, "raw_average_key_size": 20, "raw_value_size": 14534000, "raw_average_value_size": 4579, "num_data_blocks": 1875, "num_entries": 3174, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 12:57:02.239318 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1005] Compacted 1@0 + 1@1 files to L1 => 14736738 bytes
>2018-07-11 12:57:02.281113 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.281041) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.3 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.3, 15.3) out(14.1), read-write-amplify(115.4) write-amplify(54.7) OK, records in: 3665, records dropped: 491
>
>2018-07-11 12:57:02.281122 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-12:57:02.281064) EVENT_LOG_v1 {"time_micros": 1531313822281054, "job": 1005, "event": "compaction_finished", "compaction_time_micros": 216888, "output_level": 1, "num_output_files": 1, "total_output_size": 14736738, "num_input_records": 3665, "num_output_records": 3174, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 12:57:02.281255 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313822281250, "job": 1005, "event": "table_file_deletion", "file_number": 10634}
>2018-07-11 12:57:02.284225 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531313822284222, "job": 1005, "event": "table_file_deletion", "file_number": 10632}
>2018-07-11 12:57:02.284288 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 12:57:03.655438 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 12:57:21.790828 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:57:21.790851 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3992218093' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:57:21.968298 7f29149e4700  1 mon.magna102@0(leader).log v21333 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21333)
>2018-07-11 12:57:25.421269 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 12:57:25.421294 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/692416101' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 12:57:25.422570 7f29191ed700  1 mon.magna102@0(leader).log v21333 check_sub sending message to client.? 10.8.128.102:0/692416101 with 1 entries (version 21333)
>2018-07-11 12:57:25.977139 7f29149e4700  1 mon.magna102@0(leader).log v21334 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21334)
>2018-07-11 12:57:25.977172 7f29149e4700  1 mon.magna102@0(leader).log v21334 check_sub sending message to client.? 10.8.128.102:0/692416101 with 0 entries (version 21334)
>2018-07-11 12:57:30.838147 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:57:30.994431 7f29149e4700  1 mon.magna102@0(leader).log v21335 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21335)
>2018-07-11 12:57:30.994476 7f29149e4700  1 mon.magna102@0(leader).log v21335 check_sub sending message to client.? 10.8.128.102:0/692416101 with 1 entries (version 21335)
>2018-07-11 12:57:34.839968 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:57:34.986658 7f29149e4700  1 mon.magna102@0(leader).log v21336 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21336)
>2018-07-11 12:57:34.986703 7f29149e4700  1 mon.magna102@0(leader).log v21336 check_sub sending message to client.? 10.8.128.102:0/692416101 with 1 entries (version 21336)
>2018-07-11 12:57:50.846944 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:57:51.021217 7f29149e4700  1 mon.magna102@0(leader).log v21337 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21337)
>2018-07-11 12:57:51.021251 7f29149e4700  1 mon.magna102@0(leader).log v21337 check_sub sending message to client.? 10.8.128.102:0/692416101 with 1 entries (version 21337)
>2018-07-11 12:57:56.890152 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:57:57.021900 7f29149e4700  1 mon.magna102@0(leader).log v21338 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21338)
>2018-07-11 12:57:57.021946 7f29149e4700  1 mon.magna102@0(leader).log v21338 check_sub sending message to client.? 10.8.128.102:0/692416101 with 1 entries (version 21338)
>2018-07-11 12:58:03.655634 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 12:59:03.655812 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 12:59:04.879633 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 12:59:05.036110 7f29149e4700  1 mon.magna102@0(leader).log v21339 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21339)
>2018-07-11 12:59:14.884053 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 12:59:15.037010 7f29149e4700  1 mon.magna102@0(leader).log v21340 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21340)
>2018-07-11 12:59:30.891011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 6,10 (REQUEST_SLOW)
>2018-07-11 12:59:31.064468 7f29149e4700  1 mon.magna102@0(leader).log v21341 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21341)
>2018-07-11 13:00:03.655998 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:00:40.921786 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 13:00:41.071095 7f29149e4700  1 mon.magna102@0(leader).log v21342 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21342)
>2018-07-11 13:00:50.225314 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:00:50.225341 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/176689402' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:00:50.226612 7f29191ed700  1 mon.magna102@0(leader).log v21342 check_sub sending message to client.? 10.8.128.102:0/176689402 with 1 entries (version 21342)
>2018-07-11 13:00:50.346741 7f29149e4700  1 mon.magna102@0(leader).log v21343 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21343)
>2018-07-11 13:00:50.346774 7f29149e4700  1 mon.magna102@0(leader).log v21343 check_sub sending message to client.? 10.8.128.102:0/176689402 with 0 entries (version 21343)
>2018-07-11 13:00:51.346744 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 13:00:51.497130 7f29149e4700  1 mon.magna102@0(leader).log v21344 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21344)
>2018-07-11 13:00:51.497176 7f29149e4700  1 mon.magna102@0(leader).log v21344 check_sub sending message to client.? 10.8.128.102:0/176689402 with 1 entries (version 21344)
>2018-07-11 13:01:03.656198 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:01:08.073338 7f29149e4700  1 mon.magna102@0(leader).log v21345 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21345)
>2018-07-11 13:01:09.140165 7f29149e4700  1 mon.magna102@0(leader).log v21346 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21346)
>2018-07-11 13:01:12.090734 7f29149e4700  1 mon.magna102@0(leader).log v21347 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21347)
>2018-07-11 13:01:16.870497 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:01:16.870520 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1875271064' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:01:16.999509 7f29149e4700  1 mon.magna102@0(leader).log v21348 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21348)
>2018-07-11 13:01:22.100096 7f29149e4700  1 mon.magna102@0(leader).log v21349 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21349)
>2018-07-11 13:01:29.850818 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:01:29.850842 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2379552017' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:01:30.100885 7f29149e4700  1 mon.magna102@0(leader).log v21350 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21350)
>2018-07-11 13:01:31.177232 7f29149e4700  1 mon.magna102@0(leader).log v21351 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21351)
>2018-07-11 13:01:31.537812 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:01:31.537835 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/576000484' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:01:32.319336 7f29149e4700  1 mon.magna102@0(leader).log v21352 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21352)
>2018-07-11 13:01:32.848264 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:01:32.848287 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1611564098' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:01:33.452976 7f29149e4700  1 mon.magna102@0(leader).log v21353 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21353)
>2018-07-11 13:01:35.017379 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:01:35.017406 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/688121507' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:01:35.018601 7f29191ed700  1 mon.magna102@0(leader).log v21353 check_sub sending message to client.? 10.8.128.102:0/688121507 with 1 entries (version 21353)
>2018-07-11 13:01:35.136679 7f29149e4700  1 mon.magna102@0(leader).log v21354 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21354)
>2018-07-11 13:01:35.136712 7f29149e4700  1 mon.magna102@0(leader).log v21354 check_sub sending message to client.? 10.8.128.102:0/688121507 with 0 entries (version 21354)
>2018-07-11 13:01:44.087326 7f29149e4700  1 mon.magna102@0(leader).log v21355 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21355)
>2018-07-11 13:01:44.087358 7f29149e4700  1 mon.magna102@0(leader).log v21355 check_sub sending message to client.? 10.8.128.102:0/688121507 with 0 entries (version 21355)
>2018-07-11 13:01:54.088251 7f29149e4700  1 mon.magna102@0(leader).log v21356 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21356)
>2018-07-11 13:01:54.088284 7f29149e4700  1 mon.magna102@0(leader).log v21356 check_sub sending message to client.? 10.8.128.102:0/688121507 with 0 entries (version 21356)
>2018-07-11 13:02:03.656404 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:03:03.656601 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:03:41.130427 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10636. Immutable memtables: 0.
>
>2018-07-11 13:03:41.130472 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1006] Syncing log #10633
>2018-07-11 13:03:41.146920 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.130462) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:03:41.146932 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1006] Flushing memtable with next log file: 10636
>
>2018-07-11 13:03:41.146946 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221146938, "job": 1006, "event": "flush_started", "num_memtables": 1, "num_entries": 1848, "num_deletes": 251, "memory_usage": 3510032}
>2018-07-11 13:03:41.146949 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1006] Level-0 flush table #10637: started
>2018-07-11 13:03:41.222413 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221222398, "cf_name": "default", "job": 1006, "event": "table_file_creation", "file_number": 10637, "file_size": 3351488, "table_properties": {"data_size": 3324335, "index_size": 14992, "filter_size": 11237, "raw_key_size": 15111, "raw_average_key_size": 20, "raw_value_size": 3303422, "raw_average_value_size": 4512, "num_data_blocks": 451, "num_entries": 732, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:03:41.222424 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1006] Level-0 flush table #10637: 3351488 bytes OK
>2018-07-11 13:03:41.280784 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.238952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10637 started
>2018-07-11 13:03:41.280792 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.280746) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10637: memtable #1 done
>2018-07-11 13:03:41.280795 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.280761) EVENT_LOG_v1 {"time_micros": 1531314221280754, "job": 1006, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:03:41.280797 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.280773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:03:41.280803 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1006] Try to delete WAL files size 3473471, prev total WAL file size 3473471, number of live WAL files 2.
>
>2018-07-11 13:03:41.281659 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:03:41.281713 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.281706) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:03:41.281722 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1007] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:03:41.281728 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1006 Base level 0, inputs: [10637(3272KB)], [10635(14MB)]
>
>2018-07-11 13:03:41.281741 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221281731, "job": 1007, "event": "compaction_started", "files_L0": [10637], "files_L1": [10635], "score": -1, "input_data_size": 18088226}
>2018-07-11 13:03:41.466044 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1007] Generated table #10638: 3396 keys, 16168587 bytes
>2018-07-11 13:03:41.466085 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221466059, "cf_name": "default", "job": 1007, "event": "table_file_creation", "file_number": 10638, "file_size": 16168587, "table_properties": {"data_size": 16044280, "index_size": 71163, "filter_size": 52217, "raw_key_size": 68702, "raw_average_key_size": 20, "raw_value_size": 15946064, "raw_average_value_size": 4695, "num_data_blocks": 2074, "num_entries": 3396, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:03:41.499231 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1007] Compacted 1@0 + 1@1 files to L1 => 16168587 bytes
>2018-07-11 13:03:41.532749 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.532681) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.2 rd, 74.3 wr, level 1, files in(1, 1) out(1) MB in(3.2, 14.1) out(15.4), read-write-amplify(10.2) write-amplify(4.8) OK, records in: 3906, records dropped: 510
>
>2018-07-11 13:03:41.532758 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:03:41.532702) EVENT_LOG_v1 {"time_micros": 1531314221532692, "job": 1007, "event": "compaction_finished", "compaction_time_micros": 217473, "output_level": 1, "num_output_files": 1, "total_output_size": 16168587, "num_input_records": 3906, "num_output_records": 3396, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:03:41.533484 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221533479, "job": 1007, "event": "table_file_deletion", "file_number": 10637}
>2018-07-11 13:03:41.536384 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314221536381, "job": 1007, "event": "table_file_deletion", "file_number": 10635}
>2018-07-11 13:04:03.656787 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:05:03.656973 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:05:22.149868 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10639. Immutable memtables: 0.
>
>2018-07-11 13:05:22.149927 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1008] Syncing log #10636
>2018-07-11 13:05:22.166261 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.149917) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:05:22.166272 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1008] Flushing memtable with next log file: 10639
>
>2018-07-11 13:05:22.166299 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322166291, "job": 1008, "event": "flush_started", "num_memtables": 1, "num_entries": 605, "num_deletes": 250, "memory_usage": 769384}
>2018-07-11 13:05:22.166302 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1008] Level-0 flush table #10640: started
>2018-07-11 13:05:22.208445 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322208431, "cf_name": "default", "job": 1008, "event": "table_file_creation", "file_number": 10640, "file_size": 430905, "table_properties": {"data_size": 426115, "index_size": 1879, "filter_size": 1992, "raw_key_size": 6810, "raw_average_key_size": 21, "raw_value_size": 420522, "raw_average_value_size": 1343, "num_data_blocks": 59, "num_entries": 313, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:05:22.208455 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1008] Level-0 flush table #10640: 430905 bytes OK
>2018-07-11 13:05:22.258454 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.224994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10640 started
>2018-07-11 13:05:22.258462 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.258419) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10640: memtable #1 done
>2018-07-11 13:05:22.258465 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.258433) EVENT_LOG_v1 {"time_micros": 1531314322258427, "job": 1008, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:05:22.258480 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.258445) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:05:22.258487 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1008] Try to delete WAL files size 756317, prev total WAL file size 756317, number of live WAL files 2.
>
>2018-07-11 13:05:22.258781 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:05:22.258836 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.258829) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:05:22.258845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1009] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:05:22.258851 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1008 Base level 0, inputs: [10640(420KB)], [10638(15MB)]
>
>2018-07-11 13:05:22.258866 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322258856, "job": 1009, "event": "compaction_started", "files_L0": [10640], "files_L1": [10638], "score": -1, "input_data_size": 16599492}
>2018-07-11 13:05:22.442453 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1009] Generated table #10641: 3247 keys, 15184460 bytes
>2018-07-11 13:05:22.442503 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322442488, "cf_name": "default", "job": 1009, "event": "table_file_creation", "file_number": 10641, "file_size": 15184460, "table_properties": {"data_size": 15068698, "index_size": 65863, "filter_size": 48972, "raw_key_size": 65322, "raw_average_key_size": 20, "raw_value_size": 14976395, "raw_average_value_size": 4612, "num_data_blocks": 1925, "num_entries": 3247, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:05:22.475588 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1009] Compacted 1@0 + 1@1 files to L1 => 15184460 bytes
>2018-07-11 13:05:22.517446 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.517376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.6 rd, 70.1 wr, level 1, files in(1, 1) out(1) MB in(0.4, 15.4) out(14.5), read-write-amplify(73.8) write-amplify(35.2) OK, records in: 3709, records dropped: 462
>
>2018-07-11 13:05:22.517455 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:05:22.517399) EVENT_LOG_v1 {"time_micros": 1531314322517389, "job": 1009, "event": "compaction_finished", "compaction_time_micros": 216704, "output_level": 1, "num_output_files": 1, "total_output_size": 15184460, "num_input_records": 3709, "num_output_records": 3247, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:05:22.517601 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322517597, "job": 1009, "event": "table_file_deletion", "file_number": 10640}
>2018-07-11 13:05:22.520542 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314322520523, "job": 1009, "event": "table_file_deletion", "file_number": 10638}
>2018-07-11 13:05:22.520593 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:06:03.225251 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:06:03.225277 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/107096370' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:06:03.226599 7f29191ed700  1 mon.magna102@0(leader).log v21356 check_sub sending message to client.? 10.8.128.102:0/107096370 with 1 entries (version 21356)
>2018-07-11 13:06:03.657150 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:06:04.230622 7f29149e4700  1 mon.magna102@0(leader).log v21357 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21357)
>2018-07-11 13:06:04.230655 7f29149e4700  1 mon.magna102@0(leader).log v21357 check_sub sending message to client.? 10.8.128.102:0/107096370 with 0 entries (version 21357)
>2018-07-11 13:06:06.199225 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:06:06.199251 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1153669753' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:06:06.200560 7f29191ed700  1 mon.magna102@0(leader).log v21357 check_sub sending message to client.? 10.8.128.102:0/1153669753 with 1 entries (version 21357)
>2018-07-11 13:06:06.389293 7f29149e4700  1 mon.magna102@0(leader).log v21358 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21358)
>2018-07-11 13:06:06.389326 7f29149e4700  1 mon.magna102@0(leader).log v21358 check_sub sending message to client.? 10.8.128.102:0/1153669753 with 0 entries (version 21358)
>2018-07-11 13:06:21.215526 7f29149e4700  1 mon.magna102@0(leader).log v21359 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21359)
>2018-07-11 13:07:03.657343 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:08:03.657524 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:09:03.657703 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:10:03.657880 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:10:08.301194 7f29149e4700  1 mon.magna102@0(leader).log v21360 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21360)
>2018-07-11 13:10:09.376327 7f29149e4700  1 mon.magna102@0(leader).log v21361 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21361)
>2018-07-11 13:10:18.993854 7f29149e4700  1 mon.magna102@0(leader).log v21362 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21362)
>2018-07-11 13:10:20.094207 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:10:20.094240 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2484563940' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:10:20.095524 7f29191ed700  1 mon.magna102@0(leader).log v21362 check_sub sending message to client.? 10.8.128.102:0/2484563940 with 1 entries (version 21362)
>2018-07-11 13:10:21.127502 7f29149e4700  1 mon.magna102@0(leader).log v21363 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21363)
>2018-07-11 13:10:21.127522 7f29149e4700  1 mon.magna102@0(leader).log v21363 check_sub sending message to client.? 10.8.128.102:0/2484563940 with 0 entries (version 21363)
>2018-07-11 13:11:03.658057 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:11:45.376861 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10642. Immutable memtables: 0.
>
>2018-07-11 13:11:45.376906 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1010] Syncing log #10639
>2018-07-11 13:11:45.394564 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.376896) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:11:45.394575 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1010] Flushing memtable with next log file: 10642
>
>2018-07-11 13:11:45.394602 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705394594, "job": 1010, "event": "flush_started", "num_memtables": 1, "num_entries": 1656, "num_deletes": 251, "memory_usage": 3035472}
>2018-07-11 13:11:45.394605 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1010] Level-0 flush table #10643: started
>2018-07-11 13:11:45.470141 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705470127, "cf_name": "default", "job": 1010, "event": "table_file_creation", "file_number": 10643, "file_size": 2902004, "table_properties": {"data_size": 2877963, "index_size": 13246, "filter_size": 9871, "raw_key_size": 13738, "raw_average_key_size": 20, "raw_value_size": 2859218, "raw_average_value_size": 4299, "num_data_blocks": 400, "num_entries": 665, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:11:45.470152 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1010] Level-0 flush table #10643: 2902004 bytes OK
>2018-07-11 13:11:45.520150 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.486655) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10643 started
>2018-07-11 13:11:45.520158 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.520113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10643: memtable #1 done
>2018-07-11 13:11:45.520161 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.520127) EVENT_LOG_v1 {"time_micros": 1531314705520121, "job": 1010, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:11:45.520175 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.520140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:11:45.520189 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1010] Try to delete WAL files size 3002807, prev total WAL file size 3002807, number of live WAL files 2.
>
>2018-07-11 13:11:45.520925 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:11:45.521119 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 13:11:45.521126 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 76732.4 total, 930.4 interval
>Cumulative writes: 101K writes, 480K keys, 101K commit groups, 1.0 writes per commit group, ingest: 0.80 GB, 0.01 MB/s
>Cumulative WAL: 101K writes, 101K syncs, 1.00 writes per sync, written: 0.80 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1002 writes, 4533 keys, 1002 commit groups, 1.0 writes per commit group, ingest: 7.26 MB, 0.01 MB/s
>Interval WAL: 1002 writes, 1002 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.77 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       506    0.080       0      0
>  L1      1/1   14.48 MB   0.0      8.0     0.7      7.3       7.3     -0.0       0.0  10.7     72.7     66.5       112       503    0.223   1809K   217K
> Sum      2/2   17.25 MB   0.0      8.0     0.7      7.3       8.0      0.7       0.0  11.7     53.4     53.4       153      1009    0.152   1809K   217K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.9     56.3     56.4         1         8    0.146     15K   1973
>Uptime(secs): 76732.4 total, 76732.4 interval
>Flush(GB): cumulative 0.684, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.98 GB write, 0.11 MB/s write, 7.97 GB read, 0.11 MB/s read, 152.9 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.77 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       506    0.080       0      0
>  L1      1/1   14.48 MB   0.0      8.0     0.7      7.3       7.3     -0.0       0.0  10.7     72.7     66.5       112       503    0.223   1809K   217K
> Sum      2/2   17.25 MB   0.0      8.0     0.7      7.3       8.0      0.7       0.0  11.7     53.4     53.4       153      1009    0.152   1809K   217K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 76732.4 total, 76732.4 interval
>Flush(GB): cumulative 0.684, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 7.98 GB write, 0.11 MB/s write, 7.97 GB read, 0.11 MB/s read, 152.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 13:11:45.521151 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.521144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:11:45.521156 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1011] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:11:45.521162 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1010 Base level 0, inputs: [10643(2833KB)], [10641(14MB)]
>
>2018-07-11 13:11:45.521174 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705521165, "job": 1011, "event": "compaction_started", "files_L0": [10643], "files_L1": [10641], "score": -1, "input_data_size": 18086464}
>2018-07-11 13:11:45.729158 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1011] Generated table #10644: 3402 keys, 16132038 bytes
>2018-07-11 13:11:45.729221 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705729205, "cf_name": "default", "job": 1011, "event": "table_file_creation", "file_number": 10644, "file_size": 16132038, "table_properties": {"data_size": 16007817, "index_size": 71137, "filter_size": 52157, "raw_key_size": 68813, "raw_average_key_size": 20, "raw_value_size": 15909531, "raw_average_value_size": 4676, "num_data_blocks": 2073, "num_entries": 3402, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:11:45.770505 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1011] Compacted 1@0 + 1@1 files to L1 => 16132038 bytes
>2018-07-11 13:11:45.804165 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.804095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.5 rd, 64.7 wr, level 1, files in(1, 1) out(1) MB in(2.8, 14.5) out(15.4), read-write-amplify(11.8) write-amplify(5.6) OK, records in: 3912, records dropped: 510
>
>2018-07-11 13:11:45.804173 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:11:45.804116) EVENT_LOG_v1 {"time_micros": 1531314705804107, "job": 1011, "event": "compaction_finished", "compaction_time_micros": 249303, "output_level": 1, "num_output_files": 1, "total_output_size": 16132038, "num_input_records": 3912, "num_output_records": 3402, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:11:45.804787 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705804783, "job": 1011, "event": "table_file_deletion", "file_number": 10643}
>2018-07-11 13:11:45.807745 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314705807742, "job": 1011, "event": "table_file_deletion", "file_number": 10641}
>2018-07-11 13:12:03.658232 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:12:43.137800 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:12:43.137823 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2658132521' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:12:43.258425 7f29149e4700  1 mon.magna102@0(leader).log v21364 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21364)
>2018-07-11 13:12:46.090926 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:12:46.090963 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1844216287' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:12:46.092285 7f29191ed700  1 mon.magna102@0(leader).log v21364 check_sub sending message to client.? 10.8.128.102:0/1844216287 with 1 entries (version 21364)
>2018-07-11 13:12:46.376849 7f29149e4700  1 mon.magna102@0(leader).log v21365 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21365)
>2018-07-11 13:12:46.376881 7f29149e4700  1 mon.magna102@0(leader).log v21365 check_sub sending message to client.? 10.8.128.102:0/1844216287 with 0 entries (version 21365)
>2018-07-11 13:13:03.658370 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:13:42.381251 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10645. Immutable memtables: 0.
>
>2018-07-11 13:13:42.381296 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1012] Syncing log #10642
>2018-07-11 13:13:42.397794 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.381286) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:13:42.397805 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1012] Flushing memtable with next log file: 10645
>
>2018-07-11 13:13:42.397832 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822397824, "job": 1012, "event": "flush_started", "num_memtables": 1, "num_entries": 677, "num_deletes": 250, "memory_usage": 927104}
>2018-07-11 13:13:42.397835 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1012] Level-0 flush table #10646: started
>2018-07-11 13:13:42.439980 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822439966, "cf_name": "default", "job": 1012, "event": "table_file_creation", "file_number": 10646, "file_size": 528148, "table_properties": {"data_size": 522713, "index_size": 2230, "filter_size": 2286, "raw_key_size": 7143, "raw_average_key_size": 21, "raw_value_size": 516620, "raw_average_value_size": 1570, "num_data_blocks": 70, "num_entries": 329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:13:42.439990 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1012] Level-0 flush table #10646: 528148 bytes OK
>2018-07-11 13:13:42.490011 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.456522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10646 started
>2018-07-11 13:13:42.490019 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.489975) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10646: memtable #1 done
>2018-07-11 13:13:42.490021 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.489990) EVENT_LOG_v1 {"time_micros": 1531314822489983, "job": 1012, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:13:42.490023 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.490001) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:13:42.490041 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1012] Try to delete WAL files size 912443, prev total WAL file size 912443, number of live WAL files 2.
>
>2018-07-11 13:13:42.490332 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:13:42.490381 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.490374) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:13:42.490390 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1013] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:13:42.490397 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1012 Base level 0, inputs: [10646(515KB)], [10644(15MB)]
>
>2018-07-11 13:13:42.490411 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822490401, "job": 1013, "event": "compaction_started", "files_L0": [10646], "files_L1": [10644], "score": -1, "input_data_size": 16660186}
>2018-07-11 13:13:42.673942 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1013] Generated table #10647: 3275 keys, 15303665 bytes
>2018-07-11 13:13:42.673991 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822673964, "cf_name": "default", "job": 1013, "event": "table_file_creation", "file_number": 10647, "file_size": 15303665, "table_properties": {"data_size": 15186787, "index_size": 66547, "filter_size": 49404, "raw_key_size": 65891, "raw_average_key_size": 20, "raw_value_size": 15093600, "raw_average_value_size": 4608, "num_data_blocks": 1945, "num_entries": 3275, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:13:42.707219 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1013] Compacted 1@0 + 1@1 files to L1 => 15303665 bytes
>2018-07-11 13:13:42.741982 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.741912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.8 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.5, 15.4) out(14.6), read-write-amplify(60.5) write-amplify(29.0) OK, records in: 3731, records dropped: 456
>
>2018-07-11 13:13:42.741990 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:13:42.741934) EVENT_LOG_v1 {"time_micros": 1531314822741924, "job": 1013, "event": "compaction_finished", "compaction_time_micros": 216792, "output_level": 1, "num_output_files": 1, "total_output_size": 15303665, "num_input_records": 3731, "num_output_records": 3275, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:13:42.742170 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822742165, "job": 1013, "event": "table_file_deletion", "file_number": 10646}
>2018-07-11 13:13:42.745000 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531314822744997, "job": 1013, "event": "table_file_deletion", "file_number": 10644}
>2018-07-11 13:13:42.745114 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:14:03.658551 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:15:03.658740 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:16:03.658932 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:17:03.659115 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:18:03.659305 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:18:38.526104 7f29149e4700  1 mon.magna102@0(leader).log v21366 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21366)
>2018-07-11 13:18:39.592935 7f29149e4700  1 mon.magna102@0(leader).log v21367 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21367)
>2018-07-11 13:18:58.994444 7f29149e4700  1 mon.magna102@0(leader).log v21368 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21368)
>2018-07-11 13:19:03.659470 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:19:18.995887 7f29149e4700  1 mon.magna102@0(leader).log v21369 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21369)
>2018-07-11 13:19:31.416267 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 13:19:31.563952 7f29149e4700  1 mon.magna102@0(leader).log v21370 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21370)
>2018-07-11 13:19:37.455733 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:19:37.572786 7f29149e4700  1 mon.magna102@0(leader).log v21371 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21371)
>2018-07-11 13:19:47.591692 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10648. Immutable memtables: 0.
>
>2018-07-11 13:19:47.591737 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1014] Syncing log #10645
>2018-07-11 13:19:47.608251 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.591727) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:19:47.608263 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1014] Flushing memtable with next log file: 10648
>
>2018-07-11 13:19:47.608290 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187608282, "job": 1014, "event": "flush_started", "num_memtables": 1, "num_entries": 1585, "num_deletes": 251, "memory_usage": 2872512}
>2018-07-11 13:19:47.608293 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1014] Level-0 flush table #10649: started
>2018-07-11 13:19:47.667130 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187667116, "cf_name": "default", "job": 1014, "event": "table_file_creation", "file_number": 10649, "file_size": 2743984, "table_properties": {"data_size": 2721114, "index_size": 12566, "filter_size": 9380, "raw_key_size": 13299, "raw_average_key_size": 20, "raw_value_size": 2703147, "raw_average_value_size": 4197, "num_data_blocks": 380, "num_entries": 644, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:19:47.667141 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1014] Level-0 flush table #10649: 2743984 bytes OK
>2018-07-11 13:19:47.717142 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.683653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10649 started
>2018-07-11 13:19:47.717150 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.717104) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10649: memtable #1 done
>2018-07-11 13:19:47.717153 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.717119) EVENT_LOG_v1 {"time_micros": 1531315187717113, "job": 1014, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:19:47.717155 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.717132) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:19:47.717174 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1014] Try to delete WAL files size 2841345, prev total WAL file size 2841345, number of live WAL files 2.
>
>2018-07-11 13:19:47.717897 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:19:47.717960 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.717953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:19:47.717967 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1015] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:19:47.717973 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1014 Base level 0, inputs: [10649(2679KB)], [10647(14MB)]
>
>2018-07-11 13:19:47.717986 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187717976, "job": 1015, "event": "compaction_started", "files_L0": [10649], "files_L1": [10647], "score": -1, "input_data_size": 18047649}
>2018-07-11 13:19:47.909513 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1015] Generated table #10650: 3409 keys, 16081877 bytes
>2018-07-11 13:19:47.909563 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187909535, "cf_name": "default", "job": 1015, "event": "table_file_creation", "file_number": 10650, "file_size": 16081877, "table_properties": {"data_size": 15957741, "index_size": 71140, "filter_size": 52069, "raw_key_size": 68943, "raw_average_key_size": 20, "raw_value_size": 15859357, "raw_average_value_size": 4652, "num_data_blocks": 2073, "num_entries": 3409, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:19:47.942687 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1015] Compacted 1@0 + 1@1 files to L1 => 16081877 bytes
>2018-07-11 13:19:47.976221 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.976148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.3 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(2.6, 14.6) out(15.3), read-write-amplify(12.4) write-amplify(5.9) OK, records in: 3919, records dropped: 510
>
>2018-07-11 13:19:47.976230 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:19:47.976169) EVENT_LOG_v1 {"time_micros": 1531315187976160, "job": 1015, "event": "compaction_finished", "compaction_time_micros": 224687, "output_level": 1, "num_output_files": 1, "total_output_size": 16081877, "num_input_records": 3919, "num_output_records": 3409, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:19:47.976837 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187976833, "job": 1015, "event": "table_file_deletion", "file_number": 10649}
>2018-07-11 13:19:47.979796 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315187979794, "job": 1015, "event": "table_file_deletion", "file_number": 10647}
>2018-07-11 13:20:03.659648 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:20:07.578919 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10651. Immutable memtables: 0.
>
>2018-07-11 13:20:07.578988 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1016] Syncing log #10648
>2018-07-11 13:20:07.595530 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.578978) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:20:07.595543 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1016] Flushing memtable with next log file: 10651
>
>2018-07-11 13:20:07.595560 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207595550, "job": 1016, "event": "flush_started", "num_memtables": 1, "num_entries": 345, "num_deletes": 263, "memory_usage": 175456}
>2018-07-11 13:20:07.595563 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1016] Level-0 flush table #10652: started
>2018-07-11 13:20:07.629349 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207629334, "cf_name": "default", "job": 1016, "event": "table_file_creation", "file_number": 10652, "file_size": 161464, "table_properties": {"data_size": 158627, "index_size": 737, "filter_size": 1181, "raw_key_size": 4844, "raw_average_key_size": 16, "raw_value_size": 154234, "raw_average_value_size": 524, "num_data_blocks": 23, "num_entries": 294, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "263", "kMergeOperands": "0"}}
>2018-07-11 13:20:07.629359 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1016] Level-0 flush table #10652: 161464 bytes OK
>2018-07-11 13:20:07.687690 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.645838) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10652 started
>2018-07-11 13:20:07.687698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.687654) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10652: memtable #1 done
>2018-07-11 13:20:07.687701 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.687669) EVENT_LOG_v1 {"time_micros": 1531315207687663, "job": 1016, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:20:07.687702 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.687680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:20:07.687722 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1016] Try to delete WAL files size 165413, prev total WAL file size 165413, number of live WAL files 2.
>
>2018-07-11 13:20:07.687862 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:20:07.687912 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.687905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'auth .. 'auth; will stop at (end)
>
>2018-07-11 13:20:07.687922 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1017] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:20:07.687928 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1016 Base level 0, inputs: [10652(157KB)], [10650(15MB)]
>
>2018-07-11 13:20:07.687942 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207687932, "job": 1017, "event": "compaction_started", "files_L0": [10652], "files_L1": [10650], "score": -1, "input_data_size": 16243341}
>2018-07-11 13:20:07.880059 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1017] Generated table #10653: 3169 keys, 15992919 bytes
>2018-07-11 13:20:07.880108 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207880081, "cf_name": "default", "job": 1017, "event": "table_file_creation", "file_number": 10653, "file_size": 15992919, "table_properties": {"data_size": 15870494, "index_size": 70199, "filter_size": 51299, "raw_key_size": 65260, "raw_average_key_size": 20, "raw_value_size": 15775864, "raw_average_value_size": 4978, "num_data_blocks": 2037, "num_entries": 3169, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:20:07.913236 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1017] Compacted 1@0 + 1@1 files to L1 => 15992919 bytes
>2018-07-11 13:20:07.955102 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.955033) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.1 rd, 71.0 wr, level 1, files in(1, 1) out(1) MB in(0.2, 15.3) out(15.3), read-write-amplify(199.6) write-amplify(99.0) OK, records in: 3703, records dropped: 534
>
>2018-07-11 13:20:07.955111 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:20:07.955055) EVENT_LOG_v1 {"time_micros": 1531315207955045, "job": 1017, "event": "compaction_finished", "compaction_time_micros": 225278, "output_level": 1, "num_output_files": 1, "total_output_size": 15992919, "num_input_records": 3703, "num_output_records": 3169, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:20:07.955241 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207955237, "job": 1017, "event": "table_file_deletion", "file_number": 10652}
>2018-07-11 13:20:07.958105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315207958103, "job": 1017, "event": "table_file_deletion", "file_number": 10650}
>2018-07-11 13:20:07.958226 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:20:41.447490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:20:41.617692 7f29149e4700  1 mon.magna102@0(leader).log v21372 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21372)
>2018-07-11 13:20:55.453726 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:20:55.602132 7f29149e4700  1 mon.magna102@0(leader).log v21373 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21373)
>2018-07-11 13:21:03.659821 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:21:09.459651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:21:09.611518 7f29149e4700  1 mon.magna102@0(leader).log v21374 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21374)
>2018-07-11 13:21:17.547766 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:21:18.603862 7f29149e4700  1 mon.magna102@0(leader).log v21375 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21375)
>2018-07-11 13:22:02.633459 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10654. Immutable memtables: 0.
>
>2018-07-11 13:22:02.633502 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1018] Syncing log #10651
>2018-07-11 13:22:02.649971 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.633492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:22:02.649982 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1018] Flushing memtable with next log file: 10654
>
>2018-07-11 13:22:02.650008 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322650000, "job": 1018, "event": "flush_started", "num_memtables": 1, "num_entries": 686, "num_deletes": 250, "memory_usage": 964560}
>2018-07-11 13:22:02.650011 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1018] Level-0 flush table #10655: started
>2018-07-11 13:22:02.692158 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322692144, "cf_name": "default", "job": 1018, "event": "table_file_creation", "file_number": 10655, "file_size": 563061, "table_properties": {"data_size": 557425, "index_size": 2331, "filter_size": 2386, "raw_key_size": 7245, "raw_average_key_size": 21, "raw_value_size": 551187, "raw_average_value_size": 1650, "num_data_blocks": 73, "num_entries": 334, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:22:02.692168 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1018] Level-0 flush table #10655: 563061 bytes OK
>2018-07-11 13:22:02.742184 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.708703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10655 started
>2018-07-11 13:22:02.742192 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.742142) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10655: memtable #1 done
>2018-07-11 13:22:02.742195 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.742156) EVENT_LOG_v1 {"time_micros": 1531315322742150, "job": 1018, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:22:02.742197 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.742168) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:22:02.742216 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1018] Try to delete WAL files size 950010, prev total WAL file size 950010, number of live WAL files 2.
>
>2018-07-11 13:22:02.742543 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:22:02.742725 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 13:22:02.742731 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 77349.6 total, 617.2 interval
>Cumulative writes: 102K writes, 483K keys, 102K commit groups, 1.0 writes per commit group, ingest: 0.80 GB, 0.01 MB/s
>Cumulative WAL: 102K writes, 102K syncs, 1.00 writes per sync, written: 0.80 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 650 writes, 3293 keys, 650 commit groups, 1.0 writes per commit group, ingest: 4.64 MB, 0.01 MB/s
>Interval WAL: 650 writes, 650 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   549.86 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        41       510    0.080       0      0
>  L1      1/1   15.25 MB   0.0      8.0     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       113       507    0.223   1824K   219K
> Sum      2/2   15.79 MB   0.0      8.0     0.7      7.4       8.0      0.7       0.0  11.7     53.4     53.4       154      1017    0.152   1824K   219K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.9     56.8     55.5         1         8    0.145     15K   2010
>Uptime(secs): 77349.6 total, 77349.6 interval
>Flush(GB): cumulative 0.687, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.04 GB write, 0.11 MB/s write, 8.04 GB read, 0.11 MB/s read, 154.1 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   549.86 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        41       510    0.080       0      0
>  L1      1/1   15.25 MB   0.0      8.0     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       113       507    0.223   1824K   219K
> Sum      2/2   15.79 MB   0.0      8.0     0.7      7.4       8.0      0.7       0.0  11.7     53.4     53.4       154      1017    0.152   1824K   219K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 77349.6 total, 77349.6 interval
>Flush(GB): cumulative 0.687, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.04 GB write, 0.11 MB/s write, 8.04 GB read, 0.11 MB/s read, 154.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 13:22:02.742752 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.742746) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:22:02.742757 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1019] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:22:02.742762 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1018 Base level 0, inputs: [10655(549KB)], [10653(15MB)]
>
>2018-07-11 13:22:02.742774 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322742766, "job": 1019, "event": "compaction_started", "files_L0": [10655], "files_L1": [10653], "score": -1, "input_data_size": 16555980}
>2018-07-11 13:22:02.917888 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1019] Generated table #10656: 3046 keys, 15192370 bytes
>2018-07-11 13:22:02.917938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322917910, "cf_name": "default", "job": 1019, "event": "table_file_creation", "file_number": 10656, "file_size": 15192370, "table_properties": {"data_size": 15077182, "index_size": 65647, "filter_size": 48614, "raw_key_size": 62418, "raw_average_key_size": 20, "raw_value_size": 14987573, "raw_average_value_size": 4920, "num_data_blocks": 1910, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:22:02.951090 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1019] Compacted 1@0 + 1@1 files to L1 => 15192370 bytes
>2018-07-11 13:22:02.984601 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.984529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 72.9 wr, level 1, files in(1, 1) out(1) MB in(0.5, 15.3) out(14.5), read-write-amplify(56.4) write-amplify(27.0) OK, records in: 3503, records dropped: 457
>
>2018-07-11 13:22:02.984610 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:22:02.984551) EVENT_LOG_v1 {"time_micros": 1531315322984542, "job": 1019, "event": "compaction_finished", "compaction_time_micros": 208299, "output_level": 1, "num_output_files": 1, "total_output_size": 15192370, "num_input_records": 3503, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:22:02.984803 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322984799, "job": 1019, "event": "table_file_deletion", "file_number": 10655}
>2018-07-11 13:22:02.987903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315322987900, "job": 1019, "event": "table_file_deletion", "file_number": 10653}
>2018-07-11 13:22:02.988018 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:22:03.668318 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:22:21.491597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:22:21.646205 7f29149e4700  1 mon.magna102@0(leader).log v21376 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21376)
>2018-07-11 13:22:35.497927 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:22:35.655597 7f29149e4700  1 mon.magna102@0(leader).log v21377 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21377)
>2018-07-11 13:22:49.503901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:22:49.673318 7f29149e4700  1 mon.magna102@0(leader).log v21378 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21378)
>2018-07-11 13:22:57.593390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:22:58.674135 7f29149e4700  1 mon.magna102@0(leader).log v21379 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21379)
>2018-07-11 13:23:03.668507 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:23:41.120507 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:23:41.120531 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1300769430' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:23:41.236687 7f29149e4700  1 mon.magna102@0(leader).log v21380 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21380)
>2018-07-11 13:23:42.958892 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:23:42.958917 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4064462750' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:23:42.960241 7f29191ed700  1 mon.magna102@0(leader).log v21380 check_sub sending message to client.? 10.8.128.102:0/4064462750 with 1 entries (version 21380)
>2018-07-11 13:23:43.370363 7f29149e4700  1 mon.magna102@0(leader).log v21381 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21381)
>2018-07-11 13:23:43.370396 7f29149e4700  1 mon.magna102@0(leader).log v21381 check_sub sending message to client.? 10.8.128.102:0/4064462750 with 0 entries (version 21381)
>2018-07-11 13:24:03.668684 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:24:05.538106 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:24:05.689031 7f29149e4700  1 mon.magna102@0(leader).log v21382 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21382)
>2018-07-11 13:24:15.542483 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:24:15.690242 7f29149e4700  1 mon.magna102@0(leader).log v21383 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21383)
>2018-07-11 13:24:29.548403 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:24:29.716410 7f29149e4700  1 mon.magna102@0(leader).log v21384 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21384)
>2018-07-11 13:24:37.597082 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:24:37.726567 7f29149e4700  1 mon.magna102@0(leader).log v21385 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21385)
>2018-07-11 13:25:03.668863 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:25:45.582086 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:25:45.732709 7f29149e4700  1 mon.magna102@0(leader).log v21386 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21386)
>2018-07-11 13:25:45.871280 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:25:45.871302 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1327093268' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:25:46.807933 7f29149e4700  1 mon.magna102@0(leader).log v21387 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21387)
>2018-07-11 13:25:55.586531 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:25:55.742177 7f29149e4700  1 mon.magna102@0(leader).log v21388 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21388)
>2018-07-11 13:26:03.669050 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:26:09.592463 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:26:09.743317 7f29149e4700  1 mon.magna102@0(leader).log v21389 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21389)
>2018-07-11 13:26:17.643707 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:26:17.777415 7f29149e4700  1 mon.magna102@0(leader).log v21390 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21390)
>2018-07-11 13:27:03.669231 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 13:27:23.766971 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10657. Immutable memtables: 0.
>
>2018-07-11 13:27:23.767015 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1020] Syncing log #10654
>2018-07-11 13:27:23.783430 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.767005) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:27:23.783441 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1020] Flushing memtable with next log file: 10657
>
>2018-07-11 13:27:23.783468 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315643783459, "job": 1020, "event": "flush_started", "num_memtables": 1, "num_entries": 1503, "num_deletes": 251, "memory_usage": 2750912}
>2018-07-11 13:27:23.783471 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1020] Level-0 flush table #10658: started
>2018-07-11 13:27:23.842358 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315643842343, "cf_name": "default", "job": 1020, "event": "table_file_creation", "file_number": 10658, "file_size": 2613295, "table_properties": {"data_size": 2591719, "index_size": 11729, "filter_size": 8923, "raw_key_size": 12924, "raw_average_key_size": 20, "raw_value_size": 2574591, "raw_average_value_size": 4106, "num_data_blocks": 355, "num_entries": 627, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:27:23.842370 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1020] Level-0 flush table #10658: 2613295 bytes OK
>2018-07-11 13:27:23.892357 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.858856) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10658 started
>2018-07-11 13:27:23.892366 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.892319) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10658: memtable #1 done
>2018-07-11 13:27:23.892368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.892334) EVENT_LOG_v1 {"time_micros": 1531315643892328, "job": 1020, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:27:23.892385 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.892347) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:27:23.892392 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1020] Try to delete WAL files size 2721458, prev total WAL file size 2721458, number of live WAL files 2.
>
>2018-07-11 13:27:23.893078 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:27:23.893141 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:27:23.893134) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:27:23.893148 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1021] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:27:23.893154 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1020 Base level 0, inputs: [10658(2552KB)], [10656(14MB)]
>
>2018-07-11 13:27:23.893167 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315643893157, "job": 1021, "event": "compaction_started", "files_L0": [10658], "files_L1": [10656], "score": -1, "input_data_size": 17805665}
>2018-07-11 13:27:24.077623 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1021] Generated table #10659: 3163 keys, 15897382 bytes
>2018-07-11 13:27:24.077667 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315644077640, "cf_name": "default", "job": 1021, "event": "table_file_creation", "file_number": 10659, "file_size": 15897382, "table_properties": {"data_size": 15776115, "index_size": 69401, "filter_size": 50939, "raw_key_size": 65095, "raw_average_key_size": 20, "raw_value_size": 15682133, "raw_average_value_size": 4957, "num_data_blocks": 2013, "num_entries": 3163, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:27:24.110806 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1021] Compacted 1@0 + 1@1 files to L1 => 15897382 bytes
>2018-07-11 13:27:24.152677 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:27:24.152607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.8 rd, 73.0 wr, level 1, files in(1, 1) out(1) MB in(2.5, 14.5) out(15.2), read-write-amplify(12.9) write-amplify(6.1) OK, records in: 3673, records dropped: 510
>
>2018-07-11 13:27:24.152686 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:27:24.152628) EVENT_LOG_v1 {"time_micros": 1531315644152618, "job": 1021, "event": "compaction_finished", "compaction_time_micros": 217624, "output_level": 1, "num_output_files": 1, "total_output_size": 15897382, "num_input_records": 3673, "num_output_records": 3163, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:27:24.153329 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315644153325, "job": 1021, "event": "table_file_deletion", "file_number": 10658}
>2018-07-11 13:27:24.156092 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315644156089, "job": 1021, "event": "table_file_deletion", "file_number": 10656}
>2018-07-11 13:27:25.626168 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:27:25.778418 7f29149e4700  1 mon.magna102@0(leader).log v21391 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21391)
>2018-07-11 13:27:35.630384 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:27:35.779307 7f29149e4700  1 mon.magna102@0(leader).log v21392 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21392)
>2018-07-11 13:27:49.636402 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:27:49.788953 7f29149e4700  1 mon.magna102@0(leader).log v21393 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21393)
>2018-07-11 13:27:55.638972 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:27:55.789539 7f29149e4700  1 mon.magna102@0(leader).log v21394 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21394)
>2018-07-11 13:28:03.681729 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:28:29.526917 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:28:29.526942 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/546834678' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:28:29.528230 7f29191ed700  1 mon.magna102@0(leader).log v21394 check_sub sending message to client.? 10.8.128.102:0/546834678 with 1 entries (version 21394)
>2018-07-11 13:28:29.651705 7f29149e4700  1 mon.magna102@0(leader).log v21395 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21395)
>2018-07-11 13:28:29.651738 7f29149e4700  1 mon.magna102@0(leader).log v21395 check_sub sending message to client.? 10.8.128.102:0/546834678 with 0 entries (version 21395)
>2018-07-11 13:28:36.336632 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:28:36.336656 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/1916425824' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:28:36.794188 7f29149e4700  1 mon.magna102@0(leader).log v21396 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21396)
>2018-07-11 13:29:03.704399 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:29:05.670562 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:29:05.821470 7f29149e4700  1 mon.magna102@0(leader).log v21397 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21397)
>2018-07-11 13:29:19.676590 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:29:19.831045 7f29149e4700  1 mon.magna102@0(leader).log v21398 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21398)
>2018-07-11 13:29:35.683534 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:29:35.833578 7f29149e4700  1 mon.magna102@0(leader).log v21399 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21399)
>2018-07-11 13:30:03.735673 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:30:22.829096 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10660. Immutable memtables: 0.
>
>2018-07-11 13:30:22.829139 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1022] Syncing log #10657
>2018-07-11 13:30:22.845616 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.829130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:30:22.845628 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1022] Flushing memtable with next log file: 10660
>
>2018-07-11 13:30:22.845655 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315822845646, "job": 1022, "event": "flush_started", "num_memtables": 1, "num_entries": 950, "num_deletes": 250, "memory_usage": 1541648}
>2018-07-11 13:30:22.845658 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1022] Level-0 flush table #10661: started
>2018-07-11 13:30:22.896150 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315822896136, "cf_name": "default", "job": 1022, "event": "table_file_creation", "file_number": 10661, "file_size": 894976, "table_properties": {"data_size": 886949, "index_size": 3687, "filter_size": 3421, "raw_key_size": 8190, "raw_average_key_size": 21, "raw_value_size": 879079, "raw_average_value_size": 2307, "num_data_blocks": 115, "num_entries": 381, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:30:22.896160 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1022] Level-0 flush table #10661: 894976 bytes OK
>2018-07-11 13:30:22.946174 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.912669) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10661 started
>2018-07-11 13:30:22.946189 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.946138) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10661: memtable #1 done
>2018-07-11 13:30:22.946192 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.946153) EVENT_LOG_v1 {"time_micros": 1531315822946147, "job": 1022, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:30:22.946208 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.946165) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:30:22.946214 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1022] Try to delete WAL files size 1522308, prev total WAL file size 1522308, number of live WAL files 2.
>
>2018-07-11 13:30:22.946669 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:30:22.946734 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:30:22.946727) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:30:22.946740 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1023] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:30:22.946747 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1022 Base level 0, inputs: [10661(874KB)], [10659(15MB)]
>
>2018-07-11 13:30:22.946760 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315822946750, "job": 1023, "event": "compaction_started", "files_L0": [10661], "files_L1": [10659], "score": -1, "input_data_size": 16792358}
>2018-07-11 13:30:23.130251 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1023] Generated table #10662: 3119 keys, 15642075 bytes
>2018-07-11 13:30:23.130302 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315823130274, "cf_name": "default", "job": 1023, "event": "table_file_creation", "file_number": 10662, "file_size": 15642075, "table_properties": {"data_size": 15523765, "index_size": 67394, "filter_size": 49989, "raw_key_size": 63902, "raw_average_key_size": 20, "raw_value_size": 15431900, "raw_average_value_size": 4947, "num_data_blocks": 1961, "num_entries": 3119, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:30:23.163421 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1023] Compacted 1@0 + 1@1 files to L1 => 15642075 bytes
>2018-07-11 13:30:23.198197 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:30:23.198115) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.5 rd, 72.2 wr, level 1, files in(1, 1) out(1) MB in(0.9, 15.2) out(14.9), read-write-amplify(36.2) write-amplify(17.5) OK, records in: 3544, records dropped: 425
>
>2018-07-11 13:30:23.198206 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:30:23.198136) EVENT_LOG_v1 {"time_micros": 1531315823198127, "job": 1023, "event": "compaction_finished", "compaction_time_micros": 216644, "output_level": 1, "num_output_files": 1, "total_output_size": 15642075, "num_input_records": 3544, "num_output_records": 3119, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:30:23.198454 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315823198449, "job": 1023, "event": "table_file_deletion", "file_number": 10661}
>2018-07-11 13:30:23.201544 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531315823201541, "job": 1023, "event": "table_file_deletion", "file_number": 10659}
>2018-07-11 13:30:23.201661 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:30:45.715266 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:30:45.892462 7f29149e4700  1 mon.magna102@0(leader).log v21400 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21400)
>2018-07-11 13:30:59.721316 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:30:59.868467 7f29149e4700  1 mon.magna102@0(leader).log v21401 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21401)
>2018-07-11 13:31:03.751928 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:31:11.726472 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 13:31:11.877918 7f29149e4700  1 mon.magna102@0(leader).log v21402 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21402)
>2018-07-11 13:31:17.789094 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:31:17.903562 7f29149e4700  1 mon.magna102@0(leader).log v21403 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21403)
>2018-07-11 13:32:03.782706 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:32:25.759440 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:32:25.926250 7f29149e4700  1 mon.magna102@0(leader).log v21404 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21404)
>2018-07-11 13:32:39.765521 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:32:39.911938 7f29149e4700  1 mon.magna102@0(leader).log v21405 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21405)
>2018-07-11 13:32:49.429557 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:32:49.429585 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2284757840' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:32:49.430845 7f29191ed700  1 mon.magna102@0(leader).log v21405 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21405)
>2018-07-11 13:32:49.545941 7f29149e4700  1 mon.magna102@0(leader).log v21406 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21406)
>2018-07-11 13:32:49.545974 7f29149e4700  1 mon.magna102@0(leader).log v21406 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 0 entries (version 21406)
>2018-07-11 13:32:55.772723 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:32:55.921745 7f29149e4700  1 mon.magna102@0(leader).log v21407 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21407)
>2018-07-11 13:32:55.921790 7f29149e4700  1 mon.magna102@0(leader).log v21407 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21407)
>2018-07-11 13:33:03.813818 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:34:03.844306 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:34:05.804024 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:34:05.978105 7f29149e4700  1 mon.magna102@0(leader).log v21408 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21408)
>2018-07-11 13:34:05.978149 7f29149e4700  1 mon.magna102@0(leader).log v21408 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21408)
>2018-07-11 13:34:19.810170 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:34:19.962580 7f29149e4700  1 mon.magna102@0(leader).log v21409 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21409)
>2018-07-11 13:34:19.962625 7f29149e4700  1 mon.magna102@0(leader).log v21409 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21409)
>2018-07-11 13:34:33.816074 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:34:33.997029 7f29149e4700  1 mon.magna102@0(leader).log v21410 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21410)
>2018-07-11 13:34:33.997074 7f29149e4700  1 mon.magna102@0(leader).log v21410 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21410)
>2018-07-11 13:34:42.871318 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:34:42.939423 7f29149e4700  1 mon.magna102@0(leader).log v21411 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21411)
>2018-07-11 13:34:42.939468 7f29149e4700  1 mon.magna102@0(leader).log v21411 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21411)
>2018-07-11 13:34:59.958974 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10663. Immutable memtables: 0.
>
>2018-07-11 13:34:59.959019 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1024] Syncing log #10660
>2018-07-11 13:34:59.975520 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:34:59.959010) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:34:59.975531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1024] Flushing memtable with next log file: 10663
>
>2018-07-11 13:34:59.975557 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316099975549, "job": 1024, "event": "flush_started", "num_memtables": 1, "num_entries": 1325, "num_deletes": 251, "memory_usage": 2371736}
>2018-07-11 13:34:59.975561 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1024] Level-0 flush table #10664: started
>2018-07-11 13:35:00.034417 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316100034403, "cf_name": "default", "job": 1024, "event": "table_file_creation", "file_number": 10664, "file_size": 2252492, "table_properties": {"data_size": 2233804, "index_size": 10005, "filter_size": 7759, "raw_key_size": 11817, "raw_average_key_size": 20, "raw_value_size": 2218640, "raw_average_value_size": 3865, "num_data_blocks": 304, "num_entries": 574, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:35:00.034427 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1024] Level-0 flush table #10664: 2252492 bytes OK
>2018-07-11 13:35:00.084413 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.050937) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10664 started
>2018-07-11 13:35:00.084421 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.084376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10664: memtable #1 done
>2018-07-11 13:35:00.084424 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.084390) EVENT_LOG_v1 {"time_micros": 1531316100084384, "job": 1024, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:35:00.084426 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.084403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:35:00.084445 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1024] Try to delete WAL files size 2345254, prev total WAL file size 2345254, number of live WAL files 2.
>
>2018-07-11 13:35:00.085037 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:35:00.085247 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 13:35:00.085253 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 78126.9 total, 777.3 interval
>Cumulative writes: 103K writes, 487K keys, 103K commit groups, 1.0 writes per commit group, ingest: 0.81 GB, 0.01 MB/s
>Cumulative WAL: 103K writes, 103K syncs, 1.00 writes per sync, written: 0.81 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 856 writes, 3778 keys, 856 commit groups, 1.0 writes per commit group, ingest: 6.28 MB, 0.01 MB/s
>Interval WAL: 856 writes, 856 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       513    0.080       0      0
>  L1      1/1   14.92 MB   0.0      8.1     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       114       510    0.223   1835K   220K
> Sum      2/2   17.07 MB   0.0      8.1     0.7      7.4       8.1      0.7       0.0  11.7     53.4     53.4       155      1023    0.151   1835K   220K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.1     56.7     58.2         1         6    0.143     10K   1392
>Uptime(secs): 78126.9 total, 78126.9 interval
>Flush(GB): cumulative 0.693, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.09 GB write, 0.11 MB/s write, 8.09 GB read, 0.11 MB/s read, 154.9 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       513    0.080       0      0
>  L1      1/1   14.92 MB   0.0      8.1     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       114       510    0.223   1835K   220K
> Sum      2/2   17.07 MB   0.0      8.1     0.7      7.4       8.1      0.7       0.0  11.7     53.4     53.4       155      1023    0.151   1835K   220K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 78126.9 total, 78126.9 interval
>Flush(GB): cumulative 0.693, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.09 GB write, 0.11 MB/s write, 8.09 GB read, 0.11 MB/s read, 154.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 13:35:00.085277 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.085270) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:35:00.085282 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1025] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:35:00.085287 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1024 Base level 0, inputs: [10664(2199KB)], [10662(14MB)]
>
>2018-07-11 13:35:00.085299 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316100085290, "job": 1025, "event": "compaction_started", "files_L0": [10664], "files_L1": [10662], "score": -1, "input_data_size": 17894567}
>2018-07-11 13:35:00.268450 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1025] Generated table #10665: 3183 keys, 15988870 bytes
>2018-07-11 13:35:00.268494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316100268480, "cf_name": "default", "job": 1025, "event": "table_file_creation", "file_number": 10665, "file_size": 15988870, "table_properties": {"data_size": 15867369, "index_size": 69414, "filter_size": 51160, "raw_key_size": 65472, "raw_average_key_size": 20, "raw_value_size": 15773094, "raw_average_value_size": 4955, "num_data_blocks": 2013, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:35:00.301486 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1025] Compacted 1@0 + 1@1 files to L1 => 15988870 bytes
>2018-07-11 13:35:00.335135 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.335068) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.8 rd, 74.0 wr, level 1, files in(1, 1) out(1) MB in(2.1, 14.9) out(15.2), read-write-amplify(15.0) write-amplify(7.1) OK, records in: 3693, records dropped: 510
>
>2018-07-11 13:35:00.335144 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:35:00.335089) EVENT_LOG_v1 {"time_micros": 1531316100335080, "job": 1025, "event": "compaction_finished", "compaction_time_micros": 216171, "output_level": 1, "num_output_files": 1, "total_output_size": 15988870, "num_input_records": 3693, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:35:00.335666 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316100335661, "job": 1025, "event": "table_file_deletion", "file_number": 10664}
>2018-07-11 13:35:00.338537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316100338534, "job": 1025, "event": "table_file_deletion", "file_number": 10662}
>2018-07-11 13:35:03.869087 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:35:45.848360 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 13:35:45.998609 7f29149e4700  1 mon.magna102@0(leader).log v21412 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21412)
>2018-07-11 13:35:45.998655 7f29149e4700  1 mon.magna102@0(leader).log v21412 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21412)
>2018-07-11 13:36:03.856237 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 13:36:03.891413 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:36:04.025094 7f29149e4700  1 mon.magna102@0(leader).log v21413 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21413)
>2018-07-11 13:36:04.025139 7f29149e4700  1 mon.magna102@0(leader).log v21413 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21413)
>2018-07-11 13:36:15.861628 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 13:36:16.034417 7f29149e4700  1 mon.magna102@0(leader).log v21414 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21414)
>2018-07-11 13:36:16.034463 7f29149e4700  1 mon.magna102@0(leader).log v21414 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21414)
>2018-07-11 13:36:22.909496 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 13:36:23.026668 7f29149e4700  1 mon.magna102@0(leader).log v21415 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21415)
>2018-07-11 13:36:23.026714 7f29149e4700  1 mon.magna102@0(leader).log v21415 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21415)
>2018-07-11 13:36:25.866137 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 1)
>2018-07-11 13:36:26.018850 7f29149e4700  1 mon.magna102@0(leader).log v21416 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21416)
>2018-07-11 13:36:26.018896 7f29149e4700  1 mon.magna102@0(leader).log v21416 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 1 entries (version 21416)
>2018-07-11 13:36:28.994071 7f29149e4700  1 mon.magna102@0(leader).log v21417 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21417)
>2018-07-11 13:36:28.994106 7f29149e4700  1 mon.magna102@0(leader).log v21417 check_sub sending message to client.? 10.8.128.102:0/2284757840 with 0 entries (version 21417)
>2018-07-11 13:36:32.292342 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:36:32.292367 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3346696873' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:36:32.293730 7f29191ed700  1 mon.magna102@0(leader).log v21417 check_sub sending message to client.? 10.8.128.102:0/3346696873 with 1 entries (version 21417)
>2018-07-11 13:36:33.002942 7f29149e4700  1 mon.magna102@0(leader).log v21418 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21418)
>2018-07-11 13:36:33.002962 7f29149e4700  1 mon.magna102@0(leader).log v21418 check_sub sending message to client.? 10.8.128.102:0/3346696873 with 0 entries (version 21418)
>2018-07-11 13:36:39.003508 7f29149e4700  1 mon.magna102@0(leader).log v21419 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21419)
>2018-07-11 13:36:49.013930 7f29149e4700  1 mon.magna102@0(leader).log v21420 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21420)
>2018-07-11 13:36:59.023227 7f29149e4700  1 mon.magna102@0(leader).log v21421 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21421)
>2018-07-11 13:37:03.915208 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:37:09.015770 7f29149e4700  1 mon.magna102@0(leader).log v21422 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21422)
>2018-07-11 13:37:17.499798 7f29149e4700  1 mon.magna102@0(leader).log v21423 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21423)
>2018-07-11 13:37:19.633517 7f29149e4700  1 mon.magna102@0(leader).log v21424 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21424)
>2018-07-11 13:37:25.520944 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:37:25.520967 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3338738250' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:37:25.642448 7f29149e4700  1 mon.magna102@0(leader).log v21425 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21425)
>2018-07-11 13:37:27.397252 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 13:37:27.397277 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/4220131668' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 13:37:27.398499 7f29191ed700  1 mon.magna102@0(leader).log v21425 check_sub sending message to client.? 10.8.128.102:0/4220131668 with 1 entries (version 21425)
>2018-07-11 13:37:27.776156 7f29149e4700  1 mon.magna102@0(leader).log v21426 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21426)
>2018-07-11 13:37:27.776197 7f29149e4700  1 mon.magna102@0(leader).log v21426 check_sub sending message to client.? 10.8.128.102:0/4220131668 with 0 entries (version 21426)
>2018-07-11 13:38:03.946661 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:38:43.024890 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10666. Immutable memtables: 0.
>
>2018-07-11 13:38:43.024934 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1026] Syncing log #10663
>2018-07-11 13:38:43.041390 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.024924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:38:43.041401 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1026] Flushing memtable with next log file: 10666
>
>2018-07-11 13:38:43.041428 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323041419, "job": 1026, "event": "flush_started", "num_memtables": 1, "num_entries": 1152, "num_deletes": 250, "memory_usage": 1984040}
>2018-07-11 13:38:43.041431 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1026] Level-0 flush table #10667: started
>2018-07-11 13:38:43.083618 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323083605, "cf_name": "default", "job": 1026, "event": "table_file_creation", "file_number": 10667, "file_size": 1180367, "table_properties": {"data_size": 1170364, "index_size": 4789, "filter_size": 4294, "raw_key_size": 8996, "raw_average_key_size": 21, "raw_value_size": 1161130, "raw_average_value_size": 2758, "num_data_blocks": 149, "num_entries": 421, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:38:43.083628 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1026] Level-0 flush table #10667: 1180367 bytes OK
>2018-07-11 13:38:43.133631 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.100144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10667 started
>2018-07-11 13:38:43.133639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.133595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10667: memtable #1 done
>2018-07-11 13:38:43.133642 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.133610) EVENT_LOG_v1 {"time_micros": 1531316323133604, "job": 1026, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:38:43.133656 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.133621) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:38:43.133663 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1026] Try to delete WAL files size 1960915, prev total WAL file size 1960915, number of live WAL files 2.
>
>2018-07-11 13:38:43.134205 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:38:43.134256 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.134249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:38:43.134266 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1027] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:38:43.134272 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1026 Base level 0, inputs: [10667(1152KB)], [10665(15MB)]
>
>2018-07-11 13:38:43.134286 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323134276, "job": 1027, "event": "compaction_started", "files_L0": [10667], "files_L1": [10665], "score": -1, "input_data_size": 17169237}
>2018-07-11 13:38:43.326029 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1027] Generated table #10668: 3201 keys, 16169830 bytes
>2018-07-11 13:38:43.326077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323326050, "cf_name": "default", "job": 1027, "event": "table_file_creation", "file_number": 10668, "file_size": 16169830, "table_properties": {"data_size": 16048014, "index_size": 69312, "filter_size": 51577, "raw_key_size": 65569, "raw_average_key_size": 20, "raw_value_size": 15953634, "raw_average_value_size": 4983, "num_data_blocks": 2017, "num_entries": 3201, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:38:43.359219 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1027] Compacted 1@0 + 1@1 files to L1 => 16169830 bytes
>2018-07-11 13:38:43.392759 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.392692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.3 rd, 71.9 wr, level 1, files in(1, 1) out(1) MB in(1.1, 15.2) out(15.4), read-write-amplify(28.2) write-amplify(13.7) OK, records in: 3604, records dropped: 403
>
>2018-07-11 13:38:43.392767 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:38:43.392714) EVENT_LOG_v1 {"time_micros": 1531316323392704, "job": 1027, "event": "compaction_finished", "compaction_time_micros": 224918, "output_level": 1, "num_output_files": 1, "total_output_size": 16169830, "num_input_records": 3604, "num_output_records": 3201, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:38:43.393082 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323393077, "job": 1027, "event": "table_file_deletion", "file_number": 10667}
>2018-07-11 13:38:43.395874 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316323395871, "job": 1027, "event": "table_file_deletion", "file_number": 10665}
>2018-07-11 13:38:43.395993 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:39:03.970868 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:40:03.994176 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:41:04.024635 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:42:04.053768 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:42:48.200308 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10669. Immutable memtables: 0.
>
>2018-07-11 13:42:48.200354 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1028] Syncing log #10666
>2018-07-11 13:42:48.216734 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.200344) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:42:48.216745 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1028] Flushing memtable with next log file: 10669
>
>2018-07-11 13:42:48.216772 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568216764, "job": 1028, "event": "flush_started", "num_memtables": 1, "num_entries": 1117, "num_deletes": 251, "memory_usage": 1861776}
>2018-07-11 13:42:48.216775 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1028] Level-0 flush table #10670: started
>2018-07-11 13:42:48.275609 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568275583, "cf_name": "default", "job": 1028, "event": "table_file_creation", "file_number": 10670, "file_size": 1775071, "table_properties": {"data_size": 1759707, "index_size": 8140, "filter_size": 6304, "raw_key_size": 10362, "raw_average_key_size": 20, "raw_value_size": 1746893, "raw_average_value_size": 3466, "num_data_blocks": 248, "num_entries": 504, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:42:48.275620 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1028] Level-0 flush table #10670: 1775071 bytes OK
>2018-07-11 13:42:48.325636 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.292127) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10670 started
>2018-07-11 13:42:48.325644 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.325600) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10670: memtable #1 done
>2018-07-11 13:42:48.325647 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.325614) EVENT_LOG_v1 {"time_micros": 1531316568325608, "job": 1028, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:42:48.325649 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.325626) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:42:48.325668 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1028] Try to delete WAL files size 1839044, prev total WAL file size 1839044, number of live WAL files 2.
>
>2018-07-11 13:42:48.326247 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:42:48.326299 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.326292) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:42:48.326309 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1029] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:42:48.326315 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1028 Base level 0, inputs: [10670(1733KB)], [10668(15MB)]
>
>2018-07-11 13:42:48.326328 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568326319, "job": 1029, "event": "compaction_started", "files_L0": [10670], "files_L1": [10668], "score": -1, "input_data_size": 17944901}
>2018-07-11 13:42:48.526346 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1029] Generated table #10671: 3197 keys, 15995142 bytes
>2018-07-11 13:42:48.526397 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568526380, "cf_name": "default", "job": 1029, "event": "table_file_creation", "file_number": 10671, "file_size": 15995142, "table_properties": {"data_size": 15873544, "index_size": 69461, "filter_size": 51210, "raw_key_size": 65735, "raw_average_key_size": 20, "raw_value_size": 15779042, "raw_average_value_size": 4935, "num_data_blocks": 2014, "num_entries": 3197, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:42:48.559532 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1029] Compacted 1@0 + 1@1 files to L1 => 15995142 bytes
>2018-07-11 13:42:48.593030 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.592962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.0 rd, 68.6 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.4) out(15.3), read-write-amplify(19.1) write-amplify(9.0) OK, records in: 3705, records dropped: 508
>
>2018-07-11 13:42:48.593039 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:42:48.592984) EVENT_LOG_v1 {"time_micros": 1531316568592974, "job": 1029, "event": "compaction_finished", "compaction_time_micros": 233187, "output_level": 1, "num_output_files": 1, "total_output_size": 15995142, "num_input_records": 3705, "num_output_records": 3197, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:42:48.593474 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568593470, "job": 1029, "event": "table_file_deletion", "file_number": 10670}
>2018-07-11 13:42:48.596617 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316568596614, "job": 1029, "event": "table_file_deletion", "file_number": 10668}
>2018-07-11 13:43:04.077748 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:44:04.099896 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:44:57.829898 7f29149e4700  1 mon.magna102@0(leader).log v21427 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21427)
>2018-07-11 13:45:04.130385 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:46:04.160757 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:47:03.241493 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10672. Immutable memtables: 0.
>
>2018-07-11 13:47:03.241536 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1030] Syncing log #10669
>2018-07-11 13:47:03.257996 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.241526) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:47:03.258008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1030] Flushing memtable with next log file: 10672
>
>2018-07-11 13:47:03.258022 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823258014, "job": 1030, "event": "flush_started", "num_memtables": 1, "num_entries": 1152, "num_deletes": 250, "memory_usage": 1941216}
>2018-07-11 13:47:03.258024 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1030] Level-0 flush table #10673: started
>2018-07-11 13:47:03.308499 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823308485, "cf_name": "default", "job": 1030, "event": "table_file_creation", "file_number": 10673, "file_size": 1026365, "table_properties": {"data_size": 1017231, "index_size": 4363, "filter_size": 3851, "raw_key_size": 8462, "raw_average_key_size": 21, "raw_value_size": 1008675, "raw_average_value_size": 2553, "num_data_blocks": 137, "num_entries": 395, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:47:03.308509 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1030] Level-0 flush table #10673: 1026365 bytes OK
>2018-07-11 13:47:03.366907 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.325090) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10673 started
>2018-07-11 13:47:03.366915 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.366872) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10673: memtable #1 done
>2018-07-11 13:47:03.366918 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.366886) EVENT_LOG_v1 {"time_micros": 1531316823366880, "job": 1030, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:47:03.366920 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.366898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:47:03.366926 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1030] Try to delete WAL files size 1918002, prev total WAL file size 1918002, number of live WAL files 2.
>
>2018-07-11 13:47:03.367405 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:47:03.367582 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 13:47:03.367589 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 78850.2 total, 723.3 interval
>Cumulative writes: 104K writes, 491K keys, 104K commit groups, 1.0 writes per commit group, ingest: 0.81 GB, 0.01 MB/s
>Cumulative WAL: 104K writes, 104K syncs, 1.00 writes per sync, written: 0.81 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 760 writes, 3421 keys, 760 commit groups, 1.0 writes per commit group, ingest: 5.45 MB, 0.01 MB/s
>Interval WAL: 760 writes, 760 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   1002.31 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       516    0.080       0      0
>  L1      1/1   15.25 MB   0.0      8.1     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       115       513    0.223   1846K   222K
> Sum      2/2   16.23 MB   0.0      8.1     0.7      7.4       8.1      0.7       0.0  11.7     53.5     53.5       156      1029    0.151   1846K   222K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  13.1     57.7     56.8         1         6    0.146     11K   1421
>Uptime(secs): 78850.2 total, 78850.2 interval
>Flush(GB): cumulative 0.697, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.14 GB write, 0.11 MB/s write, 8.14 GB read, 0.11 MB/s read, 155.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   1002.31 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       516    0.080       0      0
>  L1      1/1   15.25 MB   0.0      8.1     0.7      7.4       7.4     -0.0       0.0  10.7     72.7     66.5       115       513    0.223   1846K   222K
> Sum      2/2   16.23 MB   0.0      8.1     0.7      7.4       8.1      0.7       0.0  11.7     53.5     53.5       156      1029    0.151   1846K   222K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 78850.2 total, 78850.2 interval
>Flush(GB): cumulative 0.697, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.14 GB write, 0.11 MB/s write, 8.14 GB read, 0.11 MB/s read, 155.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 13:47:03.367609 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.367602) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:47:03.367614 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1031] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:47:03.367619 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1030 Base level 0, inputs: [10673(1002KB)], [10671(15MB)]
>
>2018-07-11 13:47:03.367631 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823367622, "job": 1031, "event": "compaction_started", "files_L0": [10673], "files_L1": [10671], "score": -1, "input_data_size": 17021507}
>2018-07-11 13:47:03.559268 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1031] Generated table #10674: 3205 keys, 16136289 bytes
>2018-07-11 13:47:03.559305 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823559289, "cf_name": "default", "job": 1031, "event": "table_file_creation", "file_number": 10674, "file_size": 16136289, "table_properties": {"data_size": 16014363, "index_size": 69447, "filter_size": 51552, "raw_key_size": 65650, "raw_average_key_size": 20, "raw_value_size": 15919832, "raw_average_value_size": 4967, "num_data_blocks": 2021, "num_entries": 3205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:47:03.592409 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1031] Compacted 1@0 + 1@1 files to L1 => 16136289 bytes
>2018-07-11 13:47:03.634297 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.634230) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.7 rd, 71.8 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.3) out(15.4), read-write-amplify(32.3) write-amplify(15.7) OK, records in: 3592, records dropped: 387
>
>2018-07-11 13:47:03.634306 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:47:03.634252) EVENT_LOG_v1 {"time_micros": 1531316823634243, "job": 1031, "event": "compaction_finished", "compaction_time_micros": 224760, "output_level": 1, "num_output_files": 1, "total_output_size": 16136289, "num_input_records": 3592, "num_output_records": 3205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:47:03.634551 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823634547, "job": 1031, "event": "table_file_deletion", "file_number": 10673}
>2018-07-11 13:47:03.637387 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531316823637384, "job": 1031, "event": "table_file_deletion", "file_number": 10671}
>2018-07-11 13:47:03.637439 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:47:04.160936 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8759 MB, avail 861 GB
>2018-07-11 13:48:04.161135 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:48:17.491280 7f29149e4700  1 mon.magna102@0(leader).log v21428 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21428)
>2018-07-11 13:48:19.625003 7f29149e4700  1 mon.magna102@0(leader).log v21429 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21429)
>2018-07-11 13:48:29.342533 7f29149e4700  1 mon.magna102@0(leader).log v21430 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21430)
>2018-07-11 13:48:39.343393 7f29149e4700  1 mon.magna102@0(leader).log v21431 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21431)
>2018-07-11 13:48:49.344266 7f29149e4700  1 mon.magna102@0(leader).log v21432 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21432)
>2018-07-11 13:49:04.161287 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:49:09.322136 7f29149e4700  1 mon.magna102@0(leader).log v21433 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21433)
>2018-07-11 13:50:04.161459 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:50:52.386286 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10675. Immutable memtables: 0.
>
>2018-07-11 13:50:52.386332 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1032] Syncing log #10672
>2018-07-11 13:50:52.402731 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.386322) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:50:52.402742 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1032] Flushing memtable with next log file: 10675
>
>2018-07-11 13:50:52.402768 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052402760, "job": 1032, "event": "flush_started", "num_memtables": 1, "num_entries": 1109, "num_deletes": 251, "memory_usage": 1845800}
>2018-07-11 13:50:52.402772 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1032] Level-0 flush table #10676: started
>2018-07-11 13:50:52.453271 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052453255, "cf_name": "default", "job": 1032, "event": "table_file_creation", "file_number": 10676, "file_size": 1763518, "table_properties": {"data_size": 1748331, "index_size": 8011, "filter_size": 6256, "raw_key_size": 10443, "raw_average_key_size": 20, "raw_value_size": 1735533, "raw_average_value_size": 3416, "num_data_blocks": 244, "num_entries": 508, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:50:52.453281 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1032] Level-0 flush table #10676: 1763518 bytes OK
>2018-07-11 13:50:52.503284 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.469789) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10676 started
>2018-07-11 13:50:52.503292 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.503248) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10676: memtable #1 done
>2018-07-11 13:50:52.503294 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.503262) EVENT_LOG_v1 {"time_micros": 1531317052503256, "job": 1032, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:50:52.503309 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.503274) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:50:52.503316 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1032] Try to delete WAL files size 1823158, prev total WAL file size 1823158, number of live WAL files 2.
>
>2018-07-11 13:50:52.503819 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:50:52.503883 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.503876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:50:52.503890 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1033] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:50:52.503897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1032 Base level 0, inputs: [10676(1722KB)], [10674(15MB)]
>
>2018-07-11 13:50:52.503910 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052503900, "job": 1033, "event": "compaction_started", "files_L0": [10676], "files_L1": [10674], "score": -1, "input_data_size": 17899807}
>2018-07-11 13:50:52.704005 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1033] Generated table #10677: 3203 keys, 15937180 bytes
>2018-07-11 13:50:52.704054 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052704026, "cf_name": "default", "job": 1033, "event": "table_file_creation", "file_number": 10677, "file_size": 15937180, "table_properties": {"data_size": 15815713, "index_size": 69433, "filter_size": 51107, "raw_key_size": 65846, "raw_average_key_size": 20, "raw_value_size": 15721153, "raw_average_value_size": 4908, "num_data_blocks": 2013, "num_entries": 3203, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:50:52.737174 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1033] Compacted 1@0 + 1@1 files to L1 => 15937180 bytes
>2018-07-11 13:50:52.770576 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.770505) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.7 rd, 68.3 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.4) out(15.2), read-write-amplify(19.2) write-amplify(9.0) OK, records in: 3713, records dropped: 510
>
>2018-07-11 13:50:52.770585 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:50:52.770527) EVENT_LOG_v1 {"time_micros": 1531317052770517, "job": 1033, "event": "compaction_finished", "compaction_time_micros": 233246, "output_level": 1, "num_output_files": 1, "total_output_size": 15937180, "num_input_records": 3713, "num_output_records": 3203, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:50:52.771007 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052771003, "job": 1033, "event": "table_file_deletion", "file_number": 10676}
>2018-07-11 13:50:52.774119 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317052774116, "job": 1033, "event": "table_file_deletion", "file_number": 10674}
>2018-07-11 13:51:04.161648 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:52:04.161817 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 13:53:04.161993 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:54:04.162177 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:55:04.162360 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:55:23.472730 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10678. Immutable memtables: 0.
>
>2018-07-11 13:55:23.472773 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1034] Syncing log #10675
>2018-07-11 13:55:23.489241 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.472764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:55:23.489253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1034] Flushing memtable with next log file: 10678
>
>2018-07-11 13:55:23.489280 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323489272, "job": 1034, "event": "flush_started", "num_memtables": 1, "num_entries": 1200, "num_deletes": 250, "memory_usage": 2041728}
>2018-07-11 13:55:23.489282 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1034] Level-0 flush table #10679: started
>2018-07-11 13:55:23.539776 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323539763, "cf_name": "default", "job": 1034, "event": "table_file_creation", "file_number": 10679, "file_size": 1067820, "table_properties": {"data_size": 1058322, "index_size": 4589, "filter_size": 3989, "raw_key_size": 8510, "raw_average_key_size": 21, "raw_value_size": 1049585, "raw_average_value_size": 2637, "num_data_blocks": 144, "num_entries": 398, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 13:55:23.539787 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1034] Level-0 flush table #10679: 1067820 bytes OK
>2018-07-11 13:55:23.589805 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.556299) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10679 started
>2018-07-11 13:55:23.589813 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.589770) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10679: memtable #1 done
>2018-07-11 13:55:23.589816 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.589784) EVENT_LOG_v1 {"time_micros": 1531317323589778, "job": 1034, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:55:23.589818 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.589796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:55:23.589838 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1034] Try to delete WAL files size 2018130, prev total WAL file size 2018130, number of live WAL files 2.
>
>2018-07-11 13:55:23.590396 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:55:23.590448 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.590441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 13:55:23.590457 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1035] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:55:23.590464 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1034 Base level 0, inputs: [10679(1042KB)], [10677(15MB)]
>
>2018-07-11 13:55:23.590477 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323590467, "job": 1035, "event": "compaction_started", "files_L0": [10679], "files_L1": [10677], "score": -1, "input_data_size": 17005000}
>2018-07-11 13:55:23.782206 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1035] Generated table #10680: 3224 keys, 16174847 bytes
>2018-07-11 13:55:23.782250 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323782223, "cf_name": "default", "job": 1035, "event": "table_file_creation", "file_number": 10680, "file_size": 16174847, "table_properties": {"data_size": 16052272, "index_size": 69892, "filter_size": 51756, "raw_key_size": 66036, "raw_average_key_size": 20, "raw_value_size": 15957165, "raw_average_value_size": 4949, "num_data_blocks": 2034, "num_entries": 3224, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:55:23.823655 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1035] Compacted 1@0 + 1@1 files to L1 => 16174847 bytes
>2018-07-11 13:55:23.857210 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.857137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.9 rd, 69.4 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.2) out(15.4), read-write-amplify(31.1) write-amplify(15.1) OK, records in: 3601, records dropped: 377
>
>2018-07-11 13:55:23.857219 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:55:23.857158) EVENT_LOG_v1 {"time_micros": 1531317323857149, "job": 1035, "event": "compaction_finished", "compaction_time_micros": 233162, "output_level": 1, "num_output_files": 1, "total_output_size": 16174847, "num_input_records": 3601, "num_output_records": 3224, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:55:23.857509 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323857505, "job": 1035, "event": "table_file_deletion", "file_number": 10679}
>2018-07-11 13:55:23.860326 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317323860323, "job": 1035, "event": "table_file_deletion", "file_number": 10677}
>2018-07-11 13:55:23.860442 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:56:04.162539 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:57:04.162712 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:58:04.162865 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:59:04.163048 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 13:59:10.619071 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10681. Immutable memtables: 0.
>
>2018-07-11 13:59:10.619115 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1036] Syncing log #10678
>2018-07-11 13:59:10.635631 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.619105) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 13:59:10.635642 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1036] Flushing memtable with next log file: 10681
>
>2018-07-11 13:59:10.635669 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317550635660, "job": 1036, "event": "flush_started", "num_memtables": 1, "num_entries": 1054, "num_deletes": 251, "memory_usage": 1727184}
>2018-07-11 13:59:10.635673 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1036] Level-0 flush table #10682: started
>2018-07-11 13:59:10.686165 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317550686151, "cf_name": "default", "job": 1036, "event": "table_file_creation", "file_number": 10682, "file_size": 1646327, "table_properties": {"data_size": 1631970, "index_size": 7547, "filter_size": 5890, "raw_key_size": 9984, "raw_average_key_size": 20, "raw_value_size": 1619840, "raw_average_value_size": 3333, "num_data_blocks": 230, "num_entries": 486, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 13:59:10.686175 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1036] Level-0 flush table #10682: 1646327 bytes OK
>2018-07-11 13:59:10.744501 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.702690) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10682 started
>2018-07-11 13:59:10.744509 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.744465) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10682: memtable #1 done
>2018-07-11 13:59:10.744512 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.744479) EVENT_LOG_v1 {"time_micros": 1531317550744473, "job": 1036, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 13:59:10.744514 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.744491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 13:59:10.744533 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1036] Try to delete WAL files size 1705475, prev total WAL file size 1705475, number of live WAL files 2.
>
>2018-07-11 13:59:10.745018 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 13:59:10.745228 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 13:59:10.745234 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 79577.6 total, 727.4 interval
>Cumulative writes: 104K writes, 494K keys, 104K commit groups, 1.0 writes per commit group, ingest: 0.82 GB, 0.01 MB/s
>Cumulative WAL: 104K writes, 104K syncs, 1.00 writes per sync, written: 0.82 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 746 writes, 3363 keys, 746 commit groups, 1.0 writes per commit group, ingest: 5.29 MB, 0.01 MB/s
>Interval WAL: 746 writes, 746 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.57 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       519    0.080       0      0
>  L1      1/1   15.43 MB   0.0      8.2     0.7      7.5       7.5     -0.0       0.0  10.7     72.7     66.5       115       516    0.223   1857K   223K
> Sum      2/2   17.00 MB   0.0      8.2     0.7      7.5       8.2      0.7       0.0  11.7     53.5     53.5       157      1035    0.151   1857K   223K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.8     55.5     56.3         1         6    0.149     10K   1274
>Uptime(secs): 79577.6 total, 79577.6 interval
>Flush(GB): cumulative 0.701, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.19 GB write, 0.11 MB/s write, 8.18 GB read, 0.11 MB/s read, 156.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.57 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        41       519    0.080       0      0
>  L1      1/1   15.43 MB   0.0      8.2     0.7      7.5       7.5     -0.0       0.0  10.7     72.7     66.5       115       516    0.223   1857K   223K
> Sum      2/2   17.00 MB   0.0      8.2     0.7      7.5       8.2      0.7       0.0  11.7     53.5     53.5       157      1035    0.151   1857K   223K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 79577.6 total, 79577.6 interval
>Flush(GB): cumulative 0.701, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.19 GB write, 0.11 MB/s write, 8.18 GB read, 0.11 MB/s read, 156.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 13:59:10.745255 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:59:10.745249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 13:59:10.745260 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1037] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 13:59:10.745265 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1036 Base level 0, inputs: [10682(1607KB)], [10680(15MB)]
>
>2018-07-11 13:59:10.745278 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317550745268, "job": 1037, "event": "compaction_started", "files_L0": [10682], "files_L1": [10680], "score": -1, "input_data_size": 17821174}
>2018-07-11 13:59:10.936894 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1037] Generated table #10683: 3202 keys, 15892520 bytes
>2018-07-11 13:59:10.936939 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317550936912, "cf_name": "default", "job": 1037, "event": "table_file_creation", "file_number": 10683, "file_size": 15892520, "table_properties": {"data_size": 15771187, "index_size": 69396, "filter_size": 51010, "raw_key_size": 65824, "raw_average_key_size": 20, "raw_value_size": 15676666, "raw_average_value_size": 4895, "num_data_blocks": 2012, "num_entries": 3202, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 13:59:10.970085 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1037] Compacted 1@0 + 1@1 files to L1 => 15892520 bytes
>2018-07-11 13:59:11.003615 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:59:11.003545) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.3 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(1.6, 15.4) out(15.2), read-write-amplify(20.5) write-amplify(9.7) OK, records in: 3710, records dropped: 508
>
>2018-07-11 13:59:11.003624 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-13:59:11.003567) EVENT_LOG_v1 {"time_micros": 1531317551003557, "job": 1037, "event": "compaction_finished", "compaction_time_micros": 224791, "output_level": 1, "num_output_files": 1, "total_output_size": 15892520, "num_input_records": 3710, "num_output_records": 3202, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 13:59:11.004012 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317551004008, "job": 1037, "event": "table_file_deletion", "file_number": 10682}
>2018-07-11 13:59:11.007096 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317551007094, "job": 1037, "event": "table_file_deletion", "file_number": 10680}
>2018-07-11 13:59:59.658838 7f29149e4700  1 mon.magna102@0(leader).log v21434 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21434)
>2018-07-11 14:00:04.163232 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:00:09.635020 7f29149e4700  1 mon.magna102@0(leader).log v21435 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21435)
>2018-07-11 14:00:19.635880 7f29149e4700  1 mon.magna102@0(leader).log v21436 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21436)
>2018-07-11 14:00:29.637992 7f29149e4700  1 mon.magna102@0(leader).log v21437 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21437)
>2018-07-11 14:01:04.163406 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:02:04.163592 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:03:04.163769 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:03:43.688225 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10684. Immutable memtables: 0.
>
>2018-07-11 14:03:43.688268 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1038] Syncing log #10681
>2018-07-11 14:03:43.704632 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.688259) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:03:43.704644 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1038] Flushing memtable with next log file: 10684
>
>2018-07-11 14:03:43.704670 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317823704662, "job": 1038, "event": "flush_started", "num_memtables": 1, "num_entries": 1239, "num_deletes": 250, "memory_usage": 2113336}
>2018-07-11 14:03:43.704674 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1038] Level-0 flush table #10685: started
>2018-07-11 14:03:43.755164 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317823755150, "cf_name": "default", "job": 1038, "event": "table_file_creation", "file_number": 10685, "file_size": 1134608, "table_properties": {"data_size": 1124632, "index_size": 4851, "filter_size": 4205, "raw_key_size": 8825, "raw_average_key_size": 21, "raw_value_size": 1115476, "raw_average_value_size": 2700, "num_data_blocks": 152, "num_entries": 413, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:03:43.755174 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1038] Level-0 flush table #10685: 1134608 bytes OK
>2018-07-11 14:03:43.805197 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.771693) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10685 started
>2018-07-11 14:03:43.805205 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.805154) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10685: memtable #1 done
>2018-07-11 14:03:43.805208 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.805168) EVENT_LOG_v1 {"time_micros": 1531317823805162, "job": 1038, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:03:43.805222 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.805187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:03:43.805230 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1038] Try to delete WAL files size 2088120, prev total WAL file size 2088120, number of live WAL files 2.
>
>2018-07-11 14:03:43.805823 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:03:43.805884 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:03:43.805878) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:03:43.805891 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1039] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:03:43.805897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1038 Base level 0, inputs: [10685(1108KB)], [10683(15MB)]
>
>2018-07-11 14:03:43.805910 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317823805900, "job": 1039, "event": "compaction_started", "files_L0": [10685], "files_L1": [10683], "score": -1, "input_data_size": 17027128}
>2018-07-11 14:03:44.005900 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1039] Generated table #10686: 3237 keys, 16203263 bytes
>2018-07-11 14:03:44.005943 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317824005917, "cf_name": "default", "job": 1039, "event": "table_file_creation", "file_number": 10686, "file_size": 16203263, "table_properties": {"data_size": 16080234, "index_size": 70199, "filter_size": 51903, "raw_key_size": 66300, "raw_average_key_size": 20, "raw_value_size": 15984719, "raw_average_value_size": 4938, "num_data_blocks": 2043, "num_entries": 3237, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:03:44.047420 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1039] Compacted 1@0 + 1@1 files to L1 => 16203263 bytes
>2018-07-11 14:03:44.089290 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:03:44.089218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 70.5 rd, 67.1 wr, level 1, files in(1, 1) out(1) MB in(1.1, 15.2) out(15.5), read-write-amplify(29.3) write-amplify(14.3) OK, records in: 3615, records dropped: 378
>
>2018-07-11 14:03:44.089299 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:03:44.089239) EVENT_LOG_v1 {"time_micros": 1531317824089230, "job": 1039, "event": "compaction_finished", "compaction_time_micros": 241492, "output_level": 1, "num_output_files": 1, "total_output_size": 16203263, "num_input_records": 3615, "num_output_records": 3237, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:03:44.089600 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317824089595, "job": 1039, "event": "table_file_deletion", "file_number": 10685}
>2018-07-11 14:03:44.092415 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531317824092412, "job": 1039, "event": "table_file_deletion", "file_number": 10683}
>2018-07-11 14:03:44.092533 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:04:04.163961 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:05:04.164140 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:06:04.164314 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:07:04.164506 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:07:20.819074 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10687. Immutable memtables: 0.
>
>2018-07-11 14:07:20.819117 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1040] Syncing log #10684
>2018-07-11 14:07:20.835544 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.819108) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:07:20.835556 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1040] Flushing memtable with next log file: 10687
>
>2018-07-11 14:07:20.835582 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318040835574, "job": 1040, "event": "flush_started", "num_memtables": 1, "num_entries": 1019, "num_deletes": 251, "memory_usage": 1643544}
>2018-07-11 14:07:20.835585 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1040] Level-0 flush table #10688: started
>2018-07-11 14:07:20.894443 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318040894429, "cf_name": "default", "job": 1040, "event": "table_file_creation", "file_number": 10688, "file_size": 1565656, "table_properties": {"data_size": 1551874, "index_size": 7218, "filter_size": 5644, "raw_key_size": 9774, "raw_average_key_size": 20, "raw_value_size": 1540127, "raw_average_value_size": 3235, "num_data_blocks": 220, "num_entries": 476, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:07:20.894453 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1040] Level-0 flush table #10688: 1565656 bytes OK
>2018-07-11 14:07:20.944426 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.910958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10688 started
>2018-07-11 14:07:20.944434 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.944389) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10688: memtable #1 done
>2018-07-11 14:07:20.944436 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.944404) EVENT_LOG_v1 {"time_micros": 1531318040944397, "job": 1040, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:07:20.944438 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.944415) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:07:20.944458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1040] Try to delete WAL files size 1622142, prev total WAL file size 1622142, number of live WAL files 2.
>
>2018-07-11 14:07:20.944934 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:07:20.944998 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:07:20.944991) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:07:20.945005 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1041] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:07:20.945011 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1040 Base level 0, inputs: [10688(1528KB)], [10686(15MB)]
>
>2018-07-11 14:07:20.945025 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318040945015, "job": 1041, "event": "compaction_started", "files_L0": [10688], "files_L1": [10686], "score": -1, "input_data_size": 17768919}
>2018-07-11 14:07:21.136818 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1041] Generated table #10689: 3205 keys, 15869792 bytes
>2018-07-11 14:07:21.136869 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318041136841, "cf_name": "default", "job": 1041, "event": "table_file_creation", "file_number": 10689, "file_size": 15869792, "table_properties": {"data_size": 15748532, "index_size": 69366, "filter_size": 50967, "raw_key_size": 65878, "raw_average_key_size": 20, "raw_value_size": 15653983, "raw_average_value_size": 4884, "num_data_blocks": 2011, "num_entries": 3205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:07:21.170022 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1041] Compacted 1@0 + 1@1 files to L1 => 15869792 bytes
>2018-07-11 14:07:21.203549 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:07:21.203482) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.0 rd, 70.5 wr, level 1, files in(1, 1) out(1) MB in(1.5, 15.5) out(15.1), read-write-amplify(21.5) write-amplify(10.1) OK, records in: 3713, records dropped: 508
>
>2018-07-11 14:07:21.203558 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:07:21.203504) EVENT_LOG_v1 {"time_micros": 1531318041203495, "job": 1041, "event": "compaction_finished", "compaction_time_micros": 224982, "output_level": 1, "num_output_files": 1, "total_output_size": 15869792, "num_input_records": 3713, "num_output_records": 3205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:07:21.203961 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318041203957, "job": 1041, "event": "table_file_deletion", "file_number": 10688}
>2018-07-11 14:07:21.207059 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318041207056, "job": 1041, "event": "table_file_deletion", "file_number": 10686}
>2018-07-11 14:08:04.164685 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:08:29.876975 7f29149e4700  1 mon.magna102@0(leader).log v21438 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21438)
>2018-07-11 14:08:46.725098 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:08:46.903787 7f29149e4700  1 mon.magna102@0(leader).log v21439 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21439)
>2018-07-11 14:09:04.164864 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:09:50.753894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:09:50.909460 7f29149e4700  1 mon.magna102@0(leader).log v21440 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21440)
>2018-07-11 14:09:56.756423 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:09:56.918373 7f29149e4700  1 mon.magna102@0(leader).log v21441 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21441)
>2018-07-11 14:10:04.165050 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:10:16.765609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:10:16.920314 7f29149e4700  1 mon.magna102@0(leader).log v21442 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21442)
>2018-07-11 14:11:04.165233 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:11:26.797073 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:11:26.951643 7f29149e4700  1 mon.magna102@0(leader).log v21443 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21443)
>2018-07-11 14:11:30.798873 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:11:30.952115 7f29149e4700  1 mon.magna102@0(leader).log v21444 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21444)
>2018-07-11 14:11:46.805876 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 14:11:46.962935 7f29149e4700  1 mon.magna102@0(leader).log v21445 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21445)
>2018-07-11 14:11:53.863129 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:11:53.971833 7f29149e4700  1 mon.magna102@0(leader).log v21446 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21446)
>2018-07-11 14:12:03.948079 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10690. Immutable memtables: 0.
>
>2018-07-11 14:12:03.948123 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1042] Syncing log #10687
>2018-07-11 14:12:03.964467 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:12:03.948113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:12:03.964479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1042] Flushing memtable with next log file: 10690
>
>2018-07-11 14:12:03.964505 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318323964496, "job": 1042, "event": "flush_started", "num_memtables": 1, "num_entries": 1314, "num_deletes": 250, "memory_usage": 2300624}
>2018-07-11 14:12:03.964508 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1042] Level-0 flush table #10691: started
>2018-07-11 14:12:04.015007 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318324014993, "cf_name": "default", "job": 1042, "event": "table_file_creation", "file_number": 10691, "file_size": 1263811, "table_properties": {"data_size": 1252945, "index_size": 5345, "filter_size": 4601, "raw_key_size": 9230, "raw_average_key_size": 21, "raw_value_size": 1243148, "raw_average_value_size": 2871, "num_data_blocks": 167, "num_entries": 433, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:12:04.015017 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1042] Level-0 flush table #10691: 1263811 bytes OK
>2018-07-11 14:12:04.065029 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.031507) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10691 started
>2018-07-11 14:12:04.065037 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.064992) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10691: memtable #1 done
>2018-07-11 14:12:04.065039 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.065007) EVENT_LOG_v1 {"time_micros": 1531318324065000, "job": 1042, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:12:04.065054 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.065018) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:12:04.065061 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1042] Try to delete WAL files size 2274381, prev total WAL file size 2274381, number of live WAL files 2.
>
>2018-07-11 14:12:04.065697 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:12:04.065888 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 14:12:04.065894 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 80350.9 total, 773.3 interval
>Cumulative writes: 105K writes, 498K keys, 105K commit groups, 1.0 writes per commit group, ingest: 0.82 GB, 0.01 MB/s
>Cumulative WAL: 105K writes, 105K syncs, 1.00 writes per sync, written: 0.82 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 804 writes, 3572 keys, 804 commit groups, 1.0 writes per commit group, ingest: 5.71 MB, 0.01 MB/s
>Interval WAL: 804 writes, 804 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.21 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       522    0.080       0      0
>  L1      1/1   15.13 MB   0.0      8.2     0.7      7.5       7.5     -0.0       0.0  10.7     72.7     66.5       116       519    0.223   1868K   225K
> Sum      2/2   16.34 MB   0.0      8.2     0.7      7.5       8.2      0.7       0.0  11.7     53.5     53.5       158      1041    0.151   1868K   225K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  13.1     55.7     55.0         1         6    0.150     11K   1394
>Uptime(secs): 80350.9 total, 80350.9 interval
>Flush(GB): cumulative 0.704, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.23 GB write, 0.10 MB/s write, 8.23 GB read, 0.10 MB/s read, 157.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.21 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       522    0.080       0      0
>  L1      1/1   15.13 MB   0.0      8.2     0.7      7.5       7.5     -0.0       0.0  10.7     72.7     66.5       116       519    0.223   1868K   225K
> Sum      2/2   16.34 MB   0.0      8.2     0.7      7.5       8.2      0.7       0.0  11.7     53.5     53.5       158      1041    0.151   1868K   225K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 80350.9 total, 80350.9 interval
>Flush(GB): cumulative 0.704, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.23 GB write, 0.10 MB/s write, 8.23 GB read, 0.10 MB/s read, 157.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 14:12:04.065915 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.065909) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:12:04.065920 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1043] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:12:04.065925 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1042 Base level 0, inputs: [10691(1234KB)], [10689(15MB)]
>
>2018-07-11 14:12:04.065937 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318324065928, "job": 1043, "event": "compaction_started", "files_L0": [10691], "files_L1": [10689], "score": -1, "input_data_size": 17133603}
>2018-07-11 14:12:04.165388 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8831 MB, avail 861 GB
>2018-07-11 14:12:04.249058 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1043] Generated table #10692: 3265 keys, 16348683 bytes
>2018-07-11 14:12:04.249099 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318324249073, "cf_name": "default", "job": 1043, "event": "table_file_creation", "file_number": 10692, "file_size": 16348683, "table_properties": {"data_size": 16224496, "index_size": 70883, "filter_size": 52377, "raw_key_size": 66869, "raw_average_key_size": 20, "raw_value_size": 16128114, "raw_average_value_size": 4939, "num_data_blocks": 2063, "num_entries": 3265, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:12:04.282293 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1043] Compacted 1@0 + 1@1 files to L1 => 16348683 bytes
>2018-07-11 14:12:04.315816 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.315748) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 75.6 wr, level 1, files in(1, 1) out(1) MB in(1.2, 15.1) out(15.6), read-write-amplify(26.5) write-amplify(12.9) OK, records in: 3638, records dropped: 373
>
>2018-07-11 14:12:04.315825 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:12:04.315770) EVENT_LOG_v1 {"time_micros": 1531318324315760, "job": 1043, "event": "compaction_finished", "compaction_time_micros": 216338, "output_level": 1, "num_output_files": 1, "total_output_size": 16348683, "num_input_records": 3638, "num_output_records": 3265, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:12:04.316148 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318324316144, "job": 1043, "event": "table_file_deletion", "file_number": 10691}
>2018-07-11 14:12:04.318964 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318324318961, "job": 1043, "event": "table_file_deletion", "file_number": 10689}
>2018-07-11 14:12:04.319080 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:12:25.784879 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 14:12:25.784904 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3681118880' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 14:12:25.978172 7f29149e4700  1 mon.magna102@0(leader).log v21447 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21447)
>2018-07-11 14:12:27.716695 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 14:12:27.716720 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/3193289965' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 14:12:27.718034 7f29191ed700  1 mon.magna102@0(leader).log v21447 check_sub sending message to client.? 10.8.128.102:0/3193289965 with 1 entries (version 21447)
>2018-07-11 14:12:28.145190 7f29149e4700  1 mon.magna102@0(leader).log v21448 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21448)
>2018-07-11 14:12:28.145210 7f29149e4700  1 mon.magna102@0(leader).log v21448 check_sub sending message to client.? 10.8.128.102:0/3193289965 with 0 entries (version 21448)
>2018-07-11 14:13:00.840275 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:13:00.981797 7f29149e4700  1 mon.magna102@0(leader).log v21449 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21449)
>2018-07-11 14:13:04.165614 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:13:06.842829 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:13:06.998998 7f29149e4700  1 mon.magna102@0(leader).log v21450 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21450)
>2018-07-11 14:13:20.849064 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:13:21.000208 7f29149e4700  1 mon.magna102@0(leader).log v21451 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21451)
>2018-07-11 14:13:26.851571 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:13:27.010381 7f29149e4700  1 mon.magna102@0(leader).log v21452 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21452)
>2018-07-11 14:14:04.165785 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:14:34.882016 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:14:35.041449 7f29149e4700  1 mon.magna102@0(leader).log v21453 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21453)
>2018-07-11 14:14:40.884687 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:14:41.025460 7f29149e4700  1 mon.magna102@0(leader).log v21454 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21454)
>2018-07-11 14:14:54.890918 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 14:14:55.043268 7f29149e4700  1 mon.magna102@0(leader).log v21455 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21455)
>2018-07-11 14:15:03.027740 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10693. Immutable memtables: 0.
>
>2018-07-11 14:15:03.027797 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1044] Syncing log #10690
>2018-07-11 14:15:03.044088 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.027787) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:15:03.044099 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1044] Flushing memtable with next log file: 10693
>
>2018-07-11 14:15:03.044125 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503044117, "job": 1044, "event": "flush_started", "num_memtables": 1, "num_entries": 958, "num_deletes": 251, "memory_usage": 1536560}
>2018-07-11 14:15:03.044129 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1044] Level-0 flush table #10694: started
>2018-07-11 14:15:03.094626 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503094612, "cf_name": "default", "job": 1044, "event": "table_file_creation", "file_number": 10694, "file_size": 1456782, "table_properties": {"data_size": 1444020, "index_size": 6566, "filter_size": 5276, "raw_key_size": 9576, "raw_average_key_size": 20, "raw_value_size": 1432847, "raw_average_value_size": 3068, "num_data_blocks": 200, "num_entries": 467, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:15:03.094636 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1044] Level-0 flush table #10694: 1456782 bytes OK
>2018-07-11 14:15:03.152974 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.111155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10694 started
>2018-07-11 14:15:03.152982 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.152937) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10694: memtable #1 done
>2018-07-11 14:15:03.152984 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.152952) EVENT_LOG_v1 {"time_micros": 1531318503152946, "job": 1044, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:15:03.152986 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.152964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:15:03.153005 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1044] Try to delete WAL files size 1516277, prev total WAL file size 1516277, number of live WAL files 2.
>
>2018-07-11 14:15:03.153416 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:15:03.153466 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.153459) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:15:03.153475 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1045] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:15:03.153482 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1044 Base level 0, inputs: [10694(1422KB)], [10692(15MB)]
>
>2018-07-11 14:15:03.153494 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503153485, "job": 1045, "event": "compaction_started", "files_L0": [10694], "files_L1": [10692], "score": -1, "input_data_size": 17805465}
>2018-07-11 14:15:03.345387 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1045] Generated table #10695: 3222 keys, 15914938 bytes
>2018-07-11 14:15:03.345424 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503345409, "cf_name": "default", "job": 1045, "event": "table_file_creation", "file_number": 10695, "file_size": 15914938, "table_properties": {"data_size": 15793539, "index_size": 69380, "filter_size": 51092, "raw_key_size": 66198, "raw_average_key_size": 20, "raw_value_size": 15698756, "raw_average_value_size": 4872, "num_data_blocks": 2011, "num_entries": 3222, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:15:03.378548 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1045] Compacted 1@0 + 1@1 files to L1 => 15914938 bytes
>2018-07-11 14:15:03.412080 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.412012) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(1.4, 15.6) out(15.2), read-write-amplify(23.1) write-amplify(10.9) OK, records in: 3732, records dropped: 510
>
>2018-07-11 14:15:03.412089 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:15:03.412033) EVENT_LOG_v1 {"time_micros": 1531318503412024, "job": 1045, "event": "compaction_finished", "compaction_time_micros": 225035, "output_level": 1, "num_output_files": 1, "total_output_size": 15914938, "num_input_records": 3732, "num_output_records": 3222, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:15:03.412478 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503412474, "job": 1045, "event": "table_file_deletion", "file_number": 10694}
>2018-07-11 14:15:03.415563 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318503415560, "job": 1045, "event": "table_file_deletion", "file_number": 10692}
>2018-07-11 14:15:03.926929 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:15:04.096058 7f29149e4700  1 mon.magna102@0(leader).log v21456 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21456)
>2018-07-11 14:15:04.165972 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:16:04.166147 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:16:10.925167 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:16:11.068823 7f29149e4700  1 mon.magna102@0(leader).log v21457 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21457)
>2018-07-11 14:16:16.927684 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:16:17.077692 7f29149e4700  1 mon.magna102@0(leader).log v21458 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21458)
>2018-07-11 14:16:34.935836 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 14:16:35.088774 7f29149e4700  1 mon.magna102@0(leader).log v21459 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21459)
>2018-07-11 14:16:43.972513 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:16:44.072791 7f29149e4700  1 mon.magna102@0(leader).log v21460 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21460)
>2018-07-11 14:17:04.166326 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:17:44.967132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:17:45.128526 7f29149e4700  1 mon.magna102@0(leader).log v21461 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21461)
>2018-07-11 14:17:56.972400 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:17:57.121222 7f29149e4700  1 mon.magna102@0(leader).log v21462 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21462)
>2018-07-11 14:18:04.166510 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:18:10.978794 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:18:11.130621 7f29149e4700  1 mon.magna102@0(leader).log v21463 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21463)
>2018-07-11 14:19:04.166700 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:19:21.010339 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:19:21.178617 7f29149e4700  1 mon.magna102@0(leader).log v21464 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21464)
>2018-07-11 14:19:31.014809 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:19:31.187709 7f29149e4700  1 mon.magna102@0(leader).log v21465 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21465)
>2018-07-11 14:19:45.020932 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 14:19:45.172262 7f29149e4700  1 mon.magna102@0(leader).log v21466 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21466)
>2018-07-11 14:19:54.068463 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:19:54.164721 7f29149e4700  1 mon.magna102@0(leader).log v21467 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21467)
>2018-07-11 14:20:04.166888 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:20:24.177424 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10696. Immutable memtables: 0.
>
>2018-07-11 14:20:24.177468 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1046] Syncing log #10693
>2018-07-11 14:20:24.193896 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.177458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:20:24.193908 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1046] Flushing memtable with next log file: 10696
>
>2018-07-11 14:20:24.193935 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824193927, "job": 1046, "event": "flush_started", "num_memtables": 1, "num_entries": 1478, "num_deletes": 250, "memory_usage": 2669864}
>2018-07-11 14:20:24.193938 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1046] Level-0 flush table #10697: started
>2018-07-11 14:20:24.244454 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824244440, "cf_name": "default", "job": 1046, "event": "table_file_creation", "file_number": 10697, "file_size": 1477596, "table_properties": {"data_size": 1465235, "index_size": 6180, "filter_size": 5261, "raw_key_size": 9883, "raw_average_key_size": 21, "raw_value_size": 1454362, "raw_average_value_size": 3127, "num_data_blocks": 193, "num_entries": 465, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:20:24.244464 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1046] Level-0 flush table #10697: 1477596 bytes OK
>2018-07-11 14:20:24.294493 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.260995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10697 started
>2018-07-11 14:20:24.294501 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.294457) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10697: memtable #1 done
>2018-07-11 14:20:24.294504 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.294471) EVENT_LOG_v1 {"time_micros": 1531318824294465, "job": 1046, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:20:24.294505 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.294483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:20:24.294524 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1046] Try to delete WAL files size 2640917, prev total WAL file size 2640917, number of live WAL files 2.
>
>2018-07-11 14:20:24.295204 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:20:24.295257 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.295249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:20:24.295267 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1047] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:20:24.295273 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1046 Base level 0, inputs: [10697(1442KB)], [10695(15MB)]
>
>2018-07-11 14:20:24.295286 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824295277, "job": 1047, "event": "compaction_started", "files_L0": [10697], "files_L1": [10695], "score": -1, "input_data_size": 17392534}
>2018-07-11 14:20:24.503514 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1047] Generated table #10698: 3331 keys, 16728971 bytes
>2018-07-11 14:20:24.503562 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824503535, "cf_name": "default", "job": 1047, "event": "table_file_creation", "file_number": 10698, "file_size": 16728971, "table_properties": {"data_size": 16602091, "index_size": 72392, "filter_size": 53561, "raw_key_size": 68206, "raw_average_key_size": 20, "raw_value_size": 16503720, "raw_average_value_size": 4954, "num_data_blocks": 2107, "num_entries": 3331, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:20:24.536722 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1047] Compacted 1@0 + 1@1 files to L1 => 16728971 bytes
>2018-07-11 14:20:24.570234 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.570158) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 69.3 wr, level 1, files in(1, 1) out(1) MB in(1.4, 15.2) out(16.0), read-write-amplify(23.1) write-amplify(11.3) OK, records in: 3687, records dropped: 356
>
>2018-07-11 14:20:24.570243 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:20:24.570186) EVENT_LOG_v1 {"time_micros": 1531318824570170, "job": 1047, "event": "compaction_finished", "compaction_time_micros": 241421, "output_level": 1, "num_output_files": 1, "total_output_size": 16728971, "num_input_records": 3687, "num_output_records": 3331, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:20:24.570618 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824570614, "job": 1047, "event": "table_file_deletion", "file_number": 10697}
>2018-07-11 14:20:24.573569 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318824573566, "job": 1047, "event": "table_file_deletion", "file_number": 10695}
>2018-07-11 14:20:24.573686 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:20:55.052847 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:20:55.207717 7f29149e4700  1 mon.magna102@0(leader).log v21468 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21468)
>2018-07-11 14:21:04.167074 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:21:07.058083 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:21:07.208824 7f29149e4700  1 mon.magna102@0(leader).log v21469 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21469)
>2018-07-11 14:21:21.064361 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:21:21.234992 7f29149e4700  1 mon.magna102@0(leader).log v21470 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21470)
>2018-07-11 14:22:04.167248 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:22:31.095721 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:22:31.249623 7f29149e4700  1 mon.magna102@0(leader).log v21471 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21471)
>2018-07-11 14:22:41.100194 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:22:41.250460 7f29149e4700  1 mon.magna102@0(leader).log v21472 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21472)
>2018-07-11 14:22:45.242794 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10699. Immutable memtables: 0.
>
>2018-07-11 14:22:45.242840 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1048] Syncing log #10696
>2018-07-11 14:22:45.259357 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.242830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:22:45.259368 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1048] Flushing memtable with next log file: 10699
>
>2018-07-11 14:22:45.259395 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965259386, "job": 1048, "event": "flush_started", "num_memtables": 1, "num_entries": 793, "num_deletes": 251, "memory_usage": 1195016}
>2018-07-11 14:22:45.259398 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1048] Level-0 flush table #10700: started
>2018-07-11 14:22:45.309891 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965309877, "cf_name": "default", "job": 1048, "event": "table_file_creation", "file_number": 10700, "file_size": 1131219, "table_properties": {"data_size": 1121031, "index_size": 5048, "filter_size": 4220, "raw_key_size": 8534, "raw_average_key_size": 20, "raw_value_size": 1111665, "raw_average_value_size": 2665, "num_data_blocks": 154, "num_entries": 417, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:22:45.309901 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1048] Level-0 flush table #10700: 1131219 bytes OK
>2018-07-11 14:22:45.359938 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.326417) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10700 started
>2018-07-11 14:22:45.359946 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.359902) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10700: memtable #1 done
>2018-07-11 14:22:45.359948 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.359916) EVENT_LOG_v1 {"time_micros": 1531318965359910, "job": 1048, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:22:45.359950 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.359928) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:22:45.359969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1048] Try to delete WAL files size 1177868, prev total WAL file size 1177868, number of live WAL files 2.
>
>2018-07-11 14:22:45.360302 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:22:45.360482 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 14:22:45.360489 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 80992.2 total, 641.3 interval
>Cumulative writes: 106K writes, 501K keys, 106K commit groups, 1.0 writes per commit group, ingest: 0.83 GB, 0.01 MB/s
>Cumulative WAL: 106K writes, 106K syncs, 1.00 writes per sync, written: 0.83 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 702 writes, 3229 keys, 702 commit groups, 1.0 writes per commit group, ingest: 5.09 MB, 0.01 MB/s
>Interval WAL: 702 writes, 702 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.08 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       525    0.080       0      0
>  L1      1/1   15.95 MB   0.0      8.3     0.7      7.6       7.6      0.0       0.0  10.7     72.7     66.5       117       522    0.223   1879K   226K
> Sum      2/2   17.03 MB   0.0      8.3     0.7      7.6       8.3      0.7       0.0  11.7     53.5     53.5       159      1047    0.151   1879K   226K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  13.1     56.5     57.2         1         6    0.147     11K   1239
>Uptime(secs): 80992.2 total, 80992.2 interval
>Flush(GB): cumulative 0.708, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.28 GB write, 0.10 MB/s write, 8.28 GB read, 0.10 MB/s read, 158.5 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.08 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       525    0.080       0      0
>  L1      1/1   15.95 MB   0.0      8.3     0.7      7.6       7.6      0.0       0.0  10.7     72.7     66.5       117       522    0.223   1879K   226K
> Sum      2/2   17.03 MB   0.0      8.3     0.7      7.6       8.3      0.7       0.0  11.7     53.5     53.5       159      1047    0.151   1879K   226K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 80992.2 total, 80992.2 interval
>Flush(GB): cumulative 0.708, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.28 GB write, 0.10 MB/s write, 8.28 GB read, 0.10 MB/s read, 158.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 14:22:45.360511 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.360504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:22:45.360518 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1049] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:22:45.360524 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1048 Base level 0, inputs: [10700(1104KB)], [10698(15MB)]
>
>2018-07-11 14:22:45.360537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965360528, "job": 1049, "event": "compaction_started", "files_L0": [10700], "files_L1": [10698], "score": -1, "input_data_size": 17860190}
>2018-07-11 14:22:45.543978 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1049] Generated table #10701: 3238 keys, 15976264 bytes
>2018-07-11 14:22:45.544026 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965543999, "cf_name": "default", "job": 1049, "event": "table_file_creation", "file_number": 10701, "file_size": 15976264, "table_properties": {"data_size": 15854780, "index_size": 69325, "filter_size": 51232, "raw_key_size": 66493, "raw_average_key_size": 20, "raw_value_size": 15759812, "raw_average_value_size": 4867, "num_data_blocks": 2009, "num_entries": 3238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:22:45.577157 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1049] Compacted 1@0 + 1@1 files to L1 => 15976264 bytes
>2018-07-11 14:22:45.610678 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.610605) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.5 rd, 73.8 wr, level 1, files in(1, 1) out(1) MB in(1.1, 16.0) out(15.2), read-write-amplify(29.9) write-amplify(14.1) OK, records in: 3748, records dropped: 510
>
>2018-07-11 14:22:45.610686 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:22:45.610626) EVENT_LOG_v1 {"time_micros": 1531318965610617, "job": 1049, "event": "compaction_finished", "compaction_time_micros": 216602, "output_level": 1, "num_output_files": 1, "total_output_size": 15976264, "num_input_records": 3748, "num_output_records": 3238, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:22:45.610998 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965610994, "job": 1049, "event": "table_file_deletion", "file_number": 10700}
>2018-07-11 14:22:45.614133 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531318965614130, "job": 1049, "event": "table_file_deletion", "file_number": 10698}
>2018-07-11 14:22:51.104754 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 14:22:51.254660 7f29149e4700  1 mon.magna102@0(leader).log v21473 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21473)
>2018-07-11 14:22:57.107286 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:22:57.263582 7f29149e4700  1 mon.magna102@0(leader).log v21474 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21474)
>2018-07-11 14:23:04.167435 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:24:04.167602 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:24:11.140597 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:24:11.295056 7f29149e4700  1 mon.magna102@0(leader).log v21475 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21475)
>2018-07-11 14:24:17.143201 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 14:24:17.295627 7f29149e4700  1 mon.magna102@0(leader).log v21476 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21476)
>2018-07-11 14:24:27.147510 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 14:24:27.313289 7f29149e4700  1 mon.magna102@0(leader).log v21477 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21477)
>2018-07-11 14:24:34.196815 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 14:24:34.280522 7f29149e4700  1 mon.magna102@0(leader).log v21478 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21478)
>2018-07-11 14:24:40.289408 7f29149e4700  1 mon.magna102@0(leader).log v21479 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21479)
>2018-07-11 14:24:41.289449 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 1,11)
>2018-07-11 14:24:41.439835 7f29149e4700  1 mon.magna102@0(leader).log v21480 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21480)
>2018-07-11 14:24:43.565812 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 14:24:43.565838 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/810821623' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 14:24:43.567225 7f29191ed700  1 mon.magna102@0(leader).log v21480 check_sub sending message to client.? 10.8.128.102:0/810821623 with 1 entries (version 21480)
>2018-07-11 14:24:44.298484 7f29149e4700  1 mon.magna102@0(leader).log v21481 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21481)
>2018-07-11 14:24:44.298504 7f29149e4700  1 mon.magna102@0(leader).log v21481 check_sub sending message to client.? 10.8.128.102:0/810821623 with 0 entries (version 21481)
>2018-07-11 14:25:04.167767 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:25:08.498716 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 14:25:08.498739 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.102:0/2439251265' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 14:25:08.618819 7f29149e4700  1 mon.magna102@0(leader).log v21482 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21482)
>2018-07-11 14:25:14.294368 7f29149e4700  1 mon.magna102@0(leader).log v21483 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21483)
>2018-07-11 14:25:31.176714 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:25:31.329398 7f29149e4700  1 mon.magna102@0(leader).log v21484 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21484)
>2018-07-11 14:25:39.005431 7f29149e4700  1 mon.magna102@0(leader).log v21485 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21485)
>2018-07-11 14:26:04.167951 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:26:45.210371 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,6)
>2018-07-11 14:26:45.387472 7f29149e4700  1 mon.magna102@0(leader).log v21486 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21486)
>2018-07-11 14:27:01.217404 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:27:01.364973 7f29149e4700  1 mon.magna102@0(leader).log v21487 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21487)
>2018-07-11 14:27:04.168142 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:27:09.257015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:27:09.374059 7f29149e4700  1 mon.magna102@0(leader).log v21488 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21488)
>2018-07-11 14:27:11.040825 7f29149e4700  1 mon.magna102@0(leader).log v21489 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21489)
>2018-07-11 14:28:04.168330 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:28:19.252593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:28:19.422113 7f29149e4700  1 mon.magna102@0(leader).log v21490 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21490)
>2018-07-11 14:28:25.255313 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:28:25.431026 7f29149e4700  1 mon.magna102@0(leader).log v21491 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21491)
>2018-07-11 14:28:39.261471 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 14:28:39.423941 7f29149e4700  1 mon.magna102@0(leader).log v21492 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21492)
>2018-07-11 14:28:44.298777 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:28:44.366346 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10702. Immutable memtables: 0.
>
>2018-07-11 14:28:44.366391 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1050] Syncing log #10699
>2018-07-11 14:28:44.382807 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.366381) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:28:44.382818 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1050] Flushing memtable with next log file: 10702
>
>2018-07-11 14:28:44.382831 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324382824, "job": 1050, "event": "flush_started", "num_memtables": 1, "num_entries": 1668, "num_deletes": 250, "memory_usage": 3133664}
>2018-07-11 14:28:44.382834 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1050] Level-0 flush table #10703: started
>2018-07-11 14:28:44.466900 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324466885, "cf_name": "default", "job": 1050, "event": "table_file_creation", "file_number": 10703, "file_size": 1799960, "table_properties": {"data_size": 1785513, "index_size": 7318, "filter_size": 6209, "raw_key_size": 10661, "raw_average_key_size": 21, "raw_value_size": 1773288, "raw_average_value_size": 3518, "num_data_blocks": 228, "num_entries": 504, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:28:44.466911 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1050] Level-0 flush table #10703: 1799960 bytes OK
>2018-07-11 14:28:44.500384 7f29149e4700  1 mon.magna102@0(leader).log v21493 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21493)
>2018-07-11 14:28:44.541967 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.500132) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10703 started
>2018-07-11 14:28:44.541975 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.541930) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10703: memtable #1 done
>2018-07-11 14:28:44.541978 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.541945) EVENT_LOG_v1 {"time_micros": 1531319324541938, "job": 1050, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:28:44.541980 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.541957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:28:44.541987 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1050] Try to delete WAL files size 3101151, prev total WAL file size 3120475, number of live WAL files 2.
>
>2018-07-11 14:28:44.542769 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:28:44.542821 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.542814) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:28:44.542830 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1051] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:28:44.542836 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1050 Base level 0, inputs: [10703(1757KB)], [10701(15MB)]
>
>2018-07-11 14:28:44.542849 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324542839, "job": 1051, "event": "compaction_started", "files_L0": [10703], "files_L1": [10701], "score": -1, "input_data_size": 17776224}
>2018-07-11 14:28:44.751007 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1051] Generated table #10704: 3407 keys, 17243024 bytes
>2018-07-11 14:28:44.751050 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324751024, "cf_name": "default", "job": 1051, "event": "table_file_creation", "file_number": 10704, "file_size": 17243024, "table_properties": {"data_size": 17112844, "index_size": 74173, "filter_size": 55080, "raw_key_size": 69751, "raw_average_key_size": 20, "raw_value_size": 17012151, "raw_average_value_size": 4993, "num_data_blocks": 2159, "num_entries": 3407, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:28:44.784198 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1051] Compacted 1@0 + 1@1 files to L1 => 17243024 bytes
>2018-07-11 14:28:44.817724 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.817654) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.7 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.2) out(16.4), read-write-amplify(19.5) write-amplify(9.6) OK, records in: 3742, records dropped: 335
>
>2018-07-11 14:28:44.817733 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.817675) EVENT_LOG_v1 {"time_micros": 1531319324817665, "job": 1051, "event": "compaction_finished", "compaction_time_micros": 241333, "output_level": 1, "num_output_files": 1, "total_output_size": 17243024, "num_input_records": 3742, "num_output_records": 3407, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:28:44.818133 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324818129, "job": 1051, "event": "table_file_deletion", "file_number": 10703}
>2018-07-11 14:28:44.821062 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324821059, "job": 1051, "event": "table_file_deletion", "file_number": 10701}
>2018-07-11 14:28:44.821322 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10705. Immutable memtables: 0.
>
>2018-07-11 14:28:44.821362 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1052] Syncing log #10702
>2018-07-11 14:28:44.834658 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.821353) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:28:44.834670 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1052] Flushing memtable with next log file: 10705
>
>2018-07-11 14:28:44.834683 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324834676, "job": 1052, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21680}
>2018-07-11 14:28:44.834685 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1052] Level-0 flush table #10706: started
>2018-07-11 14:28:44.868481 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324868468, "cf_name": "default", "job": 1052, "event": "table_file_creation", "file_number": 10706, "file_size": 20443, "table_properties": {"data_size": 19373, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 19150, "raw_average_value_size": 2393, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:28:44.868491 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1052] Level-0 flush table #10706: 20443 bytes OK
>2018-07-11 14:28:44.918540 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.885071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10706 started
>2018-07-11 14:28:44.918548 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.918506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10706: memtable #1 done
>2018-07-11 14:28:44.918550 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.918520) EVENT_LOG_v1 {"time_micros": 1531319324918514, "job": 1052, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:28:44.918553 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.918532) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:28:44.918558 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1052] Try to delete WAL files size 19324, prev total WAL file size 19324, number of live WAL files 2.
>
>2018-07-11 14:28:44.918650 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:28:44.918700 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:44.918694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:28:44.918708 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1053] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:28:44.918714 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1052 Base level 0, inputs: [10706(19KB)], [10704(16MB)]
>
>2018-07-11 14:28:44.918727 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319324918717, "job": 1053, "event": "compaction_started", "files_L0": [10706], "files_L1": [10704], "score": -1, "input_data_size": 17263467}
>2018-07-11 14:28:45.127633 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1053] Generated table #10707: 3410 keys, 17262354 bytes
>2018-07-11 14:28:45.127682 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319325127655, "cf_name": "default", "job": 1053, "event": "table_file_creation", "file_number": 10707, "file_size": 17262354, "table_properties": {"data_size": 17132044, "index_size": 74242, "filter_size": 55141, "raw_key_size": 69812, "raw_average_key_size": 20, "raw_value_size": 17031261, "raw_average_value_size": 4994, "num_data_blocks": 2161, "num_entries": 3410, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:28:45.169156 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1053] Compacted 1@0 + 1@1 files to L1 => 17262354 bytes
>2018-07-11 14:28:45.202662 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:45.202592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.9 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(0.0, 16.4) out(16.5), read-write-amplify(1688.9) write-amplify(844.4) OK, records in: 3415, records dropped: 5
>
>2018-07-11 14:28:45.202671 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:28:45.202614) EVENT_LOG_v1 {"time_micros": 1531319325202604, "job": 1053, "event": "compaction_finished", "compaction_time_micros": 250412, "output_level": 1, "num_output_files": 1, "total_output_size": 17262354, "num_input_records": 3415, "num_output_records": 3410, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:28:45.202748 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319325202744, "job": 1053, "event": "table_file_deletion", "file_number": 10706}
>2018-07-11 14:28:45.206055 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319325206052, "job": 1053, "event": "table_file_deletion", "file_number": 10704}
>2018-07-11 14:29:04.168513 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 14:29:59.297585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:29:59.461289 7f29149e4700  1 mon.magna102@0(leader).log v21494 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21494)
>2018-07-11 14:30:01.298666 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:30:01.470016 7f29149e4700  1 mon.magna102@0(leader).log v21495 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21495)
>2018-07-11 14:30:04.168682 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 14:30:17.513301 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10708. Immutable memtables: 0.
>
>2018-07-11 14:30:17.513344 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1054] Syncing log #10705
>2018-07-11 14:30:17.529841 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.513335) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:30:17.529853 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1054] Flushing memtable with next log file: 10708
>
>2018-07-11 14:30:17.529879 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417529871, "job": 1054, "event": "flush_started", "num_memtables": 1, "num_entries": 601, "num_deletes": 251, "memory_usage": 778944}
>2018-07-11 14:30:17.529883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1054] Level-0 flush table #10709: started
>2018-07-11 14:30:17.572020 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417572006, "cf_name": "default", "job": 1054, "event": "table_file_creation", "file_number": 10709, "file_size": 735655, "table_properties": {"data_size": 728506, "index_size": 3272, "filter_size": 2958, "raw_key_size": 7343, "raw_average_key_size": 20, "raw_value_size": 721232, "raw_average_value_size": 2003, "num_data_blocks": 100, "num_entries": 360, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:30:17.572030 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1054] Level-0 flush table #10709: 735655 bytes OK
>2018-07-11 14:30:17.622063 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.588566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10709 started
>2018-07-11 14:30:17.622071 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.622027) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10709: memtable #1 done
>2018-07-11 14:30:17.622074 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.622042) EVENT_LOG_v1 {"time_micros": 1531319417622036, "job": 1054, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:30:17.622076 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.622053) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:30:17.622094 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1054] Try to delete WAL files size 765438, prev total WAL file size 765438, number of live WAL files 2.
>
>2018-07-11 14:30:17.622356 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:30:17.622405 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.622398) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:30:17.622414 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1055] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:30:17.622421 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1054 Base level 0, inputs: [10709(718KB)], [10707(16MB)]
>
>2018-07-11 14:30:17.622435 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417622425, "job": 1055, "event": "compaction_started", "files_L0": [10709], "files_L1": [10707], "score": -1, "input_data_size": 17998009}
>2018-07-11 14:30:17.822696 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1055] Generated table #10710: 3260 keys, 16073632 bytes
>2018-07-11 14:30:17.822745 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417822717, "cf_name": "default", "job": 1055, "event": "table_file_creation", "file_number": 10710, "file_size": 16073632, "table_properties": {"data_size": 15951891, "index_size": 69344, "filter_size": 51470, "raw_key_size": 66908, "raw_average_key_size": 20, "raw_value_size": 15856592, "raw_average_value_size": 4863, "num_data_blocks": 2009, "num_entries": 3260, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:30:17.864301 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1055] Compacted 1@0 + 1@1 files to L1 => 16073632 bytes
>2018-07-11 14:30:17.897826 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.897753) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.4 rd, 66.5 wr, level 1, files in(1, 1) out(1) MB in(0.7, 16.5) out(15.3), read-write-amplify(46.3) write-amplify(21.8) OK, records in: 3770, records dropped: 510
>
>2018-07-11 14:30:17.897835 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:30:17.897775) EVENT_LOG_v1 {"time_micros": 1531319417897765, "job": 1055, "event": "compaction_finished", "compaction_time_micros": 241849, "output_level": 1, "num_output_files": 1, "total_output_size": 16073632, "num_input_records": 3770, "num_output_records": 3260, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:30:17.898059 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417898054, "job": 1055, "event": "table_file_deletion", "file_number": 10709}
>2018-07-11 14:30:17.901337 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319417901334, "job": 1055, "event": "table_file_deletion", "file_number": 10707}
>2018-07-11 14:30:19.306632 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 14:30:19.481955 7f29149e4700  1 mon.magna102@0(leader).log v21496 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21496)
>2018-07-11 14:30:24.348467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:30:24.440736 7f29149e4700  1 mon.magna102@0(leader).log v21497 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21497)
>2018-07-11 14:31:04.168850 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:31:35.340271 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:31:35.497250 7f29149e4700  1 mon.magna102@0(leader).log v21498 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21498)
>2018-07-11 14:31:41.342700 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:31:41.506208 7f29149e4700  1 mon.magna102@0(leader).log v21499 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21499)
>2018-07-11 14:31:55.348916 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:31:55.508537 7f29149e4700  1 mon.magna102@0(leader).log v21500 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21500)
>2018-07-11 14:32:04.169015 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:33:04.169197 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:33:15.383976 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,6)
>2018-07-11 14:33:15.540420 7f29149e4700  1 mon.magna102@0(leader).log v21501 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21501)
>2018-07-11 14:33:31.390646 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:33:31.541657 7f29149e4700  1 mon.magna102@0(leader).log v21502 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21502)
>2018-07-11 14:33:39.425443 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:33:39.559117 7f29149e4700  1 mon.magna102@0(leader).log v21503 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21503)
>2018-07-11 14:34:04.169351 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:34:49.424920 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:34:49.573667 7f29149e4700  1 mon.magna102@0(leader).log v21504 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21504)
>2018-07-11 14:34:55.427589 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:34:55.582575 7f29149e4700  1 mon.magna102@0(leader).log v21505 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21505)
>2018-07-11 14:35:04.169509 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:35:09.433563 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 14:35:09.583647 7f29149e4700  1 mon.magna102@0(leader).log v21506 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21506)
>2018-07-11 14:35:14.466889 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:35:14.567493 7f29149e4700  1 mon.magna102@0(leader).log v21507 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21507)
>2018-07-11 14:36:04.169685 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:36:29.468981 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:36:29.615833 7f29149e4700  1 mon.magna102@0(leader).log v21508 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21508)
>2018-07-11 14:36:35.471652 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:36:35.626074 7f29149e4700  1 mon.magna102@0(leader).log v21509 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21509)
>2018-07-11 14:36:45.476061 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:36:45.626951 7f29149e4700  1 mon.magna102@0(leader).log v21510 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21510)
>2018-07-11 14:36:54.510541 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:36:54.619332 7f29149e4700  1 mon.magna102@0(leader).log v21511 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21511)
>2018-07-11 14:37:04.169859 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:37:04.595518 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10711. Immutable memtables: 0.
>
>2018-07-11 14:37:04.595560 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1056] Syncing log #10708
>2018-07-11 14:37:04.611970 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.595551) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:37:04.611982 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1056] Flushing memtable with next log file: 10711
>
>2018-07-11 14:37:04.612009 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824611988, "job": 1056, "event": "flush_started", "num_memtables": 1, "num_entries": 1804, "num_deletes": 250, "memory_usage": 3446560}
>2018-07-11 14:37:04.612012 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1056] Level-0 flush table #10712: started
>2018-07-11 14:37:04.670859 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824670844, "cf_name": "default", "job": 1056, "event": "table_file_creation", "file_number": 10712, "file_size": 1924514, "table_properties": {"data_size": 1909189, "index_size": 7812, "filter_size": 6593, "raw_key_size": 10897, "raw_average_key_size": 21, "raw_value_size": 1896437, "raw_average_value_size": 3675, "num_data_blocks": 244, "num_entries": 516, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:37:04.670869 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1056] Level-0 flush table #10712: 1924514 bytes OK
>2018-07-11 14:37:04.720857 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.687379) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10712 started
>2018-07-11 14:37:04.720865 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.720820) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10712: memtable #1 done
>2018-07-11 14:37:04.720868 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.720835) EVENT_LOG_v1 {"time_micros": 1531319824720829, "job": 1056, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:37:04.720883 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.720847) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:37:04.720889 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1056] Try to delete WAL files size 3412128, prev total WAL file size 3412128, number of live WAL files 2.
>
>2018-07-11 14:37:04.721778 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:37:04.721970 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 14:37:04.721977 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 81851.6 total, 859.4 interval
>Cumulative writes: 107K writes, 505K keys, 107K commit groups, 1.0 writes per commit group, ingest: 0.83 GB, 0.01 MB/s
>Cumulative WAL: 107K writes, 107K syncs, 1.00 writes per sync, written: 0.83 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 942 writes, 4081 keys, 942 commit groups, 1.0 writes per commit group, ingest: 6.96 MB, 0.01 MB/s
>Interval WAL: 942 writes, 942 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.84 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       529    0.080       0      0
>  L1      1/1   15.33 MB   0.0      8.3     0.7      7.6       7.6     -0.0       0.0  10.7     72.7     66.5       118       526    0.224   1894K   227K
> Sum      2/2   17.16 MB   0.0      8.3     0.7      7.6       8.3      0.7       0.0  11.7     53.5     53.5       160      1055    0.151   1894K   227K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  15.9     54.0     54.1         1         8    0.157     14K   1360
>Uptime(secs): 81851.6 total, 81851.6 interval
>Flush(GB): cumulative 0.712, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.35 GB write, 0.10 MB/s write, 8.35 GB read, 0.10 MB/s read, 159.8 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.3 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.84 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        42       529    0.080       0      0
>  L1      1/1   15.33 MB   0.0      8.3     0.7      7.6       7.6     -0.0       0.0  10.7     72.7     66.5       118       526    0.224   1894K   227K
> Sum      2/2   17.16 MB   0.0      8.3     0.7      7.6       8.3      0.7       0.0  11.7     53.5     53.5       160      1055    0.151   1894K   227K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 81851.6 total, 81851.6 interval
>Flush(GB): cumulative 0.712, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.35 GB write, 0.10 MB/s write, 8.35 GB read, 0.10 MB/s read, 159.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 14:37:04.722000 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.721993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:37:04.722005 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1057] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:37:04.722010 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1056 Base level 0, inputs: [10712(1879KB)], [10710(15MB)]
>
>2018-07-11 14:37:04.722022 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824722013, "job": 1057, "event": "compaction_started", "files_L0": [10712], "files_L1": [10710], "score": -1, "input_data_size": 17998146}
>2018-07-11 14:37:04.921661 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1057] Generated table #10713: 3465 keys, 17629555 bytes
>2018-07-11 14:37:04.921706 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824921679, "cf_name": "default", "job": 1057, "event": "table_file_creation", "file_number": 10713, "file_size": 17629555, "table_properties": {"data_size": 17496854, "index_size": 75542, "filter_size": 56232, "raw_key_size": 70930, "raw_average_key_size": 20, "raw_value_size": 17394376, "raw_average_value_size": 5020, "num_data_blocks": 2199, "num_entries": 3465, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:37:04.954856 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1057] Compacted 1@0 + 1@1 files to L1 => 17629555 bytes
>2018-07-11 14:37:04.988300 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.988230) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 77.3 rd, 75.7 wr, level 1, files in(1, 1) out(1) MB in(1.8, 15.3) out(16.8), read-write-amplify(18.5) write-amplify(9.2) OK, records in: 3776, records dropped: 311
>
>2018-07-11 14:37:04.988309 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:37:04.988251) EVENT_LOG_v1 {"time_micros": 1531319824988242, "job": 1057, "event": "compaction_finished", "compaction_time_micros": 232795, "output_level": 1, "num_output_files": 1, "total_output_size": 17629555, "num_input_records": 3776, "num_output_records": 3465, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:37:04.988763 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824988759, "job": 1057, "event": "table_file_deletion", "file_number": 10712}
>2018-07-11 14:37:04.991906 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319824991902, "job": 1057, "event": "table_file_deletion", "file_number": 10710}
>2018-07-11 14:37:04.992020 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:38:03.677849 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10714. Immutable memtables: 0.
>
>2018-07-11 14:38:03.677892 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1058] Syncing log #10711
>2018-07-11 14:38:03.694279 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.677883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:38:03.694291 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1058] Flushing memtable with next log file: 10714
>
>2018-07-11 14:38:03.694317 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319883694309, "job": 1058, "event": "flush_started", "num_memtables": 1, "num_entries": 466, "num_deletes": 251, "memory_usage": 481440}
>2018-07-11 14:38:03.694320 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1058] Level-0 flush table #10715: started
>2018-07-11 14:38:03.736620 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319883736606, "cf_name": "default", "job": 1058, "event": "table_file_creation", "file_number": 10715, "file_size": 451321, "table_properties": {"data_size": 446319, "index_size": 2021, "filter_size": 2062, "raw_key_size": 6456, "raw_average_key_size": 20, "raw_value_size": 440576, "raw_average_value_size": 1385, "num_data_blocks": 62, "num_entries": 318, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:38:03.736630 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1058] Level-0 flush table #10715: 451321 bytes OK
>2018-07-11 14:38:03.786531 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.753143) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10715 started
>2018-07-11 14:38:03.786539 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.786496) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10715: memtable #1 done
>2018-07-11 14:38:03.786542 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.786511) EVENT_LOG_v1 {"time_micros": 1531319883786505, "job": 1058, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:38:03.786557 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.786522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:38:03.786565 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1058] Try to delete WAL files size 470393, prev total WAL file size 470393, number of live WAL files 2.
>
>2018-07-11 14:38:03.786739 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:38:03.786789 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:38:03.786782) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:38:03.786799 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1059] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:38:03.786805 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1058 Base level 0, inputs: [10715(440KB)], [10713(16MB)]
>
>2018-07-11 14:38:03.786818 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319883786808, "job": 1059, "event": "compaction_started", "files_L0": [10715], "files_L1": [10713], "score": -1, "input_data_size": 18080876}
>2018-07-11 14:38:03.979022 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1059] Generated table #10716: 3275 keys, 16142505 bytes
>2018-07-11 14:38:03.979066 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319883979039, "cf_name": "default", "job": 1059, "event": "table_file_creation", "file_number": 10716, "file_size": 16142505, "table_properties": {"data_size": 16020545, "index_size": 69386, "filter_size": 51647, "raw_key_size": 67190, "raw_average_key_size": 20, "raw_value_size": 15925013, "raw_average_value_size": 4862, "num_data_blocks": 2010, "num_entries": 3275, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:38:04.012211 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1059] Compacted 1@0 + 1@1 files to L1 => 16142505 bytes
>2018-07-11 14:38:04.045749 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:38:04.045681) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.2 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.8) out(15.4), read-write-amplify(75.8) write-amplify(35.8) OK, records in: 3783, records dropped: 508
>
>2018-07-11 14:38:04.045758 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:38:04.045702) EVENT_LOG_v1 {"time_micros": 1531319884045693, "job": 1059, "event": "compaction_finished", "compaction_time_micros": 225379, "output_level": 1, "num_output_files": 1, "total_output_size": 16142505, "num_input_records": 3783, "num_output_records": 3275, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:38:04.045921 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319884045916, "job": 1059, "event": "table_file_deletion", "file_number": 10715}
>2018-07-11 14:38:04.049243 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531319884049240, "job": 1059, "event": "table_file_deletion", "file_number": 10713}
>2018-07-11 14:38:04.170040 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8759 MB, avail 861 GB
>2018-07-11 14:38:05.511609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:38:05.663159 7f29149e4700  1 mon.magna102@0(leader).log v21512 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21512)
>2018-07-11 14:38:11.514047 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:38:11.663737 7f29149e4700  1 mon.magna102@0(leader).log v21513 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21513)
>2018-07-11 14:38:25.520310 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 14:38:25.674643 7f29149e4700  1 mon.magna102@0(leader).log v21514 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21514)
>2018-07-11 14:38:31.522815 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:38:31.683611 7f29149e4700  1 mon.magna102@0(leader).log v21515 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21515)
>2018-07-11 14:39:04.170231 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:39:45.555406 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,6)
>2018-07-11 14:39:45.707102 7f29149e4700  1 mon.magna102@0(leader).log v21516 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21516)
>2018-07-11 14:40:04.170420 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:40:05.564109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:40:05.717090 7f29149e4700  1 mon.magna102@0(leader).log v21517 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21517)
>2018-07-11 14:41:04.170592 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 14:41:19.596293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:41:19.748703 7f29149e4700  1 mon.magna102@0(leader).log v21518 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21518)
>2018-07-11 14:41:25.599036 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:41:25.749274 7f29149e4700  1 mon.magna102@0(leader).log v21519 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21519)
>2018-07-11 14:41:39.604986 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 14:41:39.750402 7f29149e4700  1 mon.magna102@0(leader).log v21520 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21520)
>2018-07-11 14:41:44.633560 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,6 (REQUEST_SLOW)
>2018-07-11 14:41:44.710912 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10717. Immutable memtables: 0.
>
>2018-07-11 14:41:44.752287 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.710969) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:41:44.752300 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1060] Flushing memtable with next log file: 10717
>
>2018-07-11 14:41:44.752314 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320104752307, "job": 1060, "event": "flush_started", "num_memtables": 1, "num_entries": 1347, "num_deletes": 500, "memory_usage": 1898664}
>2018-07-11 14:41:44.752317 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1060] Level-0 flush table #10718: started
>2018-07-11 14:41:44.819002 7f29149e4700  1 mon.magna102@0(leader).log v21521 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21521)
>2018-07-11 14:41:44.844427 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320104844412, "cf_name": "default", "job": 1060, "event": "table_file_creation", "file_number": 10718, "file_size": 1797784, "table_properties": {"data_size": 1781995, "index_size": 7945, "filter_size": 6924, "raw_key_size": 15645, "raw_average_key_size": 20, "raw_value_size": 1765999, "raw_average_value_size": 2335, "num_data_blocks": 242, "num_entries": 756, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 14:41:44.844437 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1060] Level-0 flush table #10718: 1797784 bytes OK
>2018-07-11 14:41:44.894595 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.861190) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10718 started
>2018-07-11 14:41:44.894603 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.894558) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10718: memtable #1 done
>2018-07-11 14:41:44.894606 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.894572) EVENT_LOG_v1 {"time_micros": 1531320104894566, "job": 1060, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:41:44.894608 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.894585) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:41:44.894627 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1060] Try to delete WAL files size 1869126, prev total WAL file size 1888237, number of live WAL files 2.
>
>2018-07-11 14:41:44.895153 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:41:44.895228 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:44.895221) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 14:41:44.895238 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1061] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:41:44.895244 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1060 Base level 0, inputs: [10718(1755KB)], [10716(15MB)]
>
>2018-07-11 14:41:44.895257 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320104895248, "job": 1061, "event": "compaction_started", "files_L0": [10718], "files_L1": [10716], "score": -1, "input_data_size": 17940289}
>2018-07-11 14:41:45.086983 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1061] Generated table #10719: 3023 keys, 15525482 bytes
>2018-07-11 14:41:45.087034 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105087005, "cf_name": "default", "job": 1061, "event": "table_file_creation", "file_number": 10719, "file_size": 15525482, "table_properties": {"data_size": 15406720, "index_size": 68109, "filter_size": 49726, "raw_key_size": 62129, "raw_average_key_size": 20, "raw_value_size": 15315648, "raw_average_value_size": 5066, "num_data_blocks": 1985, "num_entries": 3023, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:41:45.128469 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1061] Compacted 1@0 + 1@1 files to L1 => 15525482 bytes
>2018-07-11 14:41:45.162030 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.161964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.9 rd, 66.6 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.4) out(14.8), read-write-amplify(18.6) write-amplify(8.6) OK, records in: 4031, records dropped: 1008
>
>2018-07-11 14:41:45.162039 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.161985) EVENT_LOG_v1 {"time_micros": 1531320105161975, "job": 1061, "event": "compaction_finished", "compaction_time_micros": 233196, "output_level": 1, "num_output_files": 1, "total_output_size": 15525482, "num_input_records": 4031, "num_output_records": 3023, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:41:45.162488 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105162484, "job": 1061, "event": "table_file_deletion", "file_number": 10718}
>2018-07-11 14:41:45.165630 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105165627, "job": 1061, "event": "table_file_deletion", "file_number": 10716}
>2018-07-11 14:41:45.165970 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10720. Immutable memtables: 0.
>
>2018-07-11 14:41:45.166010 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1062] Syncing log #10717
>2018-07-11 14:41:45.178993 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.166001) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:41:45.179004 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1062] Flushing memtable with next log file: 10720
>
>2018-07-11 14:41:45.179030 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105179022, "job": 1062, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 21448}
>2018-07-11 14:41:45.179033 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1062] Level-0 flush table #10721: started
>2018-07-11 14:41:45.221125 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105221112, "cf_name": "default", "job": 1062, "event": "table_file_creation", "file_number": 10721, "file_size": 20230, "table_properties": {"data_size": 19160, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 18938, "raw_average_value_size": 2367, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:41:45.221135 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1062] Level-0 flush table #10721: 20230 bytes OK
>2018-07-11 14:41:45.271210 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.237605) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10721 started
>2018-07-11 14:41:45.271218 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.271169) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10721: memtable #1 done
>2018-07-11 14:41:45.271220 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.271189) EVENT_LOG_v1 {"time_micros": 1531320105271177, "job": 1062, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:41:45.271235 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.271201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:41:45.271241 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1062] Try to delete WAL files size 19111, prev total WAL file size 19111, number of live WAL files 2.
>
>2018-07-11 14:41:45.271339 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:41:45.271384 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.271378) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 14:41:45.271392 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1063] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:41:45.271398 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1062 Base level 0, inputs: [10721(19KB)], [10719(14MB)]
>
>2018-07-11 14:41:45.271412 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105271401, "job": 1063, "event": "compaction_started", "files_L0": [10721], "files_L1": [10719], "score": -1, "input_data_size": 15545712}
>2018-07-11 14:41:45.463662 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1063] Generated table #10722: 3026 keys, 15544594 bytes
>2018-07-11 14:41:45.463710 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105463683, "cf_name": "default", "job": 1063, "event": "table_file_creation", "file_number": 10722, "file_size": 15544594, "table_properties": {"data_size": 15425707, "index_size": 68177, "filter_size": 49783, "raw_key_size": 62190, "raw_average_key_size": 20, "raw_value_size": 15334546, "raw_average_value_size": 5067, "num_data_blocks": 1987, "num_entries": 3026, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:41:45.496844 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1063] Compacted 1@0 + 1@1 files to L1 => 15544594 bytes
>2018-07-11 14:41:45.530333 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.530265) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.0 rd, 69.0 wr, level 1, files in(1, 1) out(1) MB in(0.0, 14.8) out(14.8), read-write-amplify(1536.8) write-amplify(768.4) OK, records in: 3031, records dropped: 5
>
>2018-07-11 14:41:45.530342 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:41:45.530287) EVENT_LOG_v1 {"time_micros": 1531320105530277, "job": 1063, "event": "compaction_finished", "compaction_time_micros": 225416, "output_level": 1, "num_output_files": 1, "total_output_size": 15544594, "num_input_records": 3031, "num_output_records": 3026, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:41:45.530436 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105530432, "job": 1063, "event": "table_file_deletion", "file_number": 10721}
>2018-07-11 14:41:45.533320 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320105533317, "job": 1063, "event": "table_file_deletion", "file_number": 10719}
>2018-07-11 14:41:45.889298 7f29149e4700  1 mon.magna102@0(leader).log v21522 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21522)
>2018-07-11 14:41:49.677427 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 14:41:49.789999 7f29149e4700  1 mon.magna102@0(leader).log v21523 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21523)
>2018-07-11 14:41:51.610468 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 14:41:51.756970 7f29149e4700  1 mon.magna102@0(leader).log v21524 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21524)
>2018-07-11 14:42:04.170756 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:42:17.760232 7f29149e4700  1 mon.magna102@0(leader).log v21525 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21525)
>2018-07-11 14:43:04.170885 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:44:04.171030 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:45:04.171220 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:45:24.809132 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10723. Immutable memtables: 0.
>
>2018-07-11 14:45:24.809175 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1064] Syncing log #10720
>2018-07-11 14:45:24.825507 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.809165) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:45:24.825518 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1064] Flushing memtable with next log file: 10723
>
>2018-07-11 14:45:24.825532 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320324825525, "job": 1064, "event": "flush_started", "num_memtables": 1, "num_entries": 1057, "num_deletes": 250, "memory_usage": 1737848}
>2018-07-11 14:45:24.825535 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1064] Level-0 flush table #10724: started
>2018-07-11 14:45:24.867854 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320324867840, "cf_name": "default", "job": 1064, "event": "table_file_creation", "file_number": 10724, "file_size": 952831, "table_properties": {"data_size": 944271, "index_size": 4022, "filter_size": 3619, "raw_key_size": 8305, "raw_average_key_size": 21, "raw_value_size": 936069, "raw_average_value_size": 2418, "num_data_blocks": 126, "num_entries": 387, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:45:24.867864 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1064] Level-0 flush table #10724: 952831 bytes OK
>2018-07-11 14:45:24.917856 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.884368) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10724 started
>2018-07-11 14:45:24.917864 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.917819) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10724: memtable #1 done
>2018-07-11 14:45:24.917866 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.917834) EVENT_LOG_v1 {"time_micros": 1531320324917828, "job": 1064, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:45:24.917868 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.917845) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:45:24.917887 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1064] Try to delete WAL files size 1715745, prev total WAL file size 1715745, number of live WAL files 2.
>
>2018-07-11 14:45:24.918362 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:45:24.918412 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:24.918405) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:45:24.918421 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1065] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:45:24.918427 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1064 Base level 0, inputs: [10724(930KB)], [10722(14MB)]
>
>2018-07-11 14:45:24.918440 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320324918431, "job": 1065, "event": "compaction_started", "files_L0": [10724], "files_L1": [10722], "score": -1, "input_data_size": 16497425}
>2018-07-11 14:45:25.110278 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1065] Generated table #10725: 3009 keys, 15491150 bytes
>2018-07-11 14:45:25.110316 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320325110301, "cf_name": "default", "job": 1065, "event": "table_file_creation", "file_number": 10725, "file_size": 15491150, "table_properties": {"data_size": 15373499, "index_size": 67219, "filter_size": 49505, "raw_key_size": 61574, "raw_average_key_size": 20, "raw_value_size": 15283331, "raw_average_value_size": 5079, "num_data_blocks": 1966, "num_entries": 3009, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:45:25.143397 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1065] Compacted 1@0 + 1@1 files to L1 => 15491150 bytes
>2018-07-11 14:45:25.176944 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:25.176873) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.3 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.8) out(14.8), read-write-amplify(33.6) write-amplify(16.3) OK, records in: 3413, records dropped: 404
>
>2018-07-11 14:45:25.176953 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:25.176894) EVENT_LOG_v1 {"time_micros": 1531320325176885, "job": 1065, "event": "compaction_finished", "compaction_time_micros": 224940, "output_level": 1, "num_output_files": 1, "total_output_size": 15491150, "num_input_records": 3413, "num_output_records": 3009, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:45:25.177247 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320325177243, "job": 1065, "event": "table_file_deletion", "file_number": 10724}
>2018-07-11 14:45:25.180309 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320325180306, "job": 1065, "event": "table_file_deletion", "file_number": 10722}
>2018-07-11 14:45:25.180376 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:45:51.847732 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10726. Immutable memtables: 0.
>
>2018-07-11 14:45:51.847774 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1066] Syncing log #10723
>2018-07-11 14:45:51.864314 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.847764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:45:51.864325 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1066] Flushing memtable with next log file: 10726
>
>2018-07-11 14:45:51.864351 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320351864332, "job": 1066, "event": "flush_started", "num_memtables": 1, "num_entries": 354, "num_deletes": 251, "memory_usage": 228496}
>2018-07-11 14:45:51.864357 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1066] Level-0 flush table #10727: started
>2018-07-11 14:45:51.906497 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320351906485, "cf_name": "default", "job": 1066, "event": "table_file_creation", "file_number": 10727, "file_size": 213585, "table_properties": {"data_size": 210382, "index_size": 970, "filter_size": 1314, "raw_key_size": 5784, "raw_average_key_size": 20, "raw_value_size": 205852, "raw_average_value_size": 719, "num_data_blocks": 30, "num_entries": 286, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:45:51.906508 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1066] Level-0 flush table #10727: 213585 bytes OK
>2018-07-11 14:45:51.956526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.923043) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10727 started
>2018-07-11 14:45:51.956534 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.956492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10727: memtable #1 done
>2018-07-11 14:45:51.956537 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.956506) EVENT_LOG_v1 {"time_micros": 1531320351956500, "job": 1066, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:45:51.956538 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.956517) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:45:51.956557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1066] Try to delete WAL files size 219087, prev total WAL file size 219087, number of live WAL files 2.
>
>2018-07-11 14:45:51.956692 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:45:51.956741 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:51.956734) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:45:51.956751 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1067] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:45:51.956757 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1066 Base level 0, inputs: [10727(208KB)], [10725(14MB)]
>
>2018-07-11 14:45:51.956770 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320351956761, "job": 1067, "event": "compaction_started", "files_L0": [10727], "files_L1": [10725], "score": -1, "input_data_size": 15704735}
>2018-07-11 14:45:52.132296 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1067] Generated table #10728: 2787 keys, 13732746 bytes
>2018-07-11 14:45:52.132334 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320352132318, "cf_name": "default", "job": 1067, "event": "table_file_creation", "file_number": 10728, "file_size": 13732746, "table_properties": {"data_size": 13627773, "index_size": 59947, "filter_size": 44099, "raw_key_size": 57162, "raw_average_key_size": 20, "raw_value_size": 13545791, "raw_average_value_size": 4860, "num_data_blocks": 1744, "num_entries": 2787, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:45:52.173792 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1067] Compacted 1@0 + 1@1 files to L1 => 13732746 bytes
>2018-07-11 14:45:52.207254 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:52.207186) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 72.4 rd, 63.3 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.8) out(13.1), read-write-amplify(137.8) write-amplify(64.3) OK, records in: 3295, records dropped: 508
>
>2018-07-11 14:45:52.207262 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:45:52.207209) EVENT_LOG_v1 {"time_micros": 1531320352207199, "job": 1067, "event": "compaction_finished", "compaction_time_micros": 217005, "output_level": 1, "num_output_files": 1, "total_output_size": 13732746, "num_input_records": 3295, "num_output_records": 2787, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:45:52.207389 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320352207385, "job": 1067, "event": "table_file_deletion", "file_number": 10727}
>2018-07-11 14:45:52.210246 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320352210243, "job": 1067, "event": "table_file_deletion", "file_number": 10725}
>2018-07-11 14:46:04.171402 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:47:04.171579 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:48:04.171755 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:49:04.171941 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:50:04.172113 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:51:04.172310 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:52:04.172478 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:53:04.172658 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:53:45.023489 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10729. Immutable memtables: 0.
>
>2018-07-11 14:53:45.023531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1068] Syncing log #10726
>2018-07-11 14:53:45.039953 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.023522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:53:45.039965 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1068] Flushing memtable with next log file: 10729
>
>2018-07-11 14:53:45.039979 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825039971, "job": 1068, "event": "flush_started", "num_memtables": 1, "num_entries": 1907, "num_deletes": 250, "memory_usage": 3536232}
>2018-07-11 14:53:45.039981 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1068] Level-0 flush table #10730: started
>2018-07-11 14:53:45.098806 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825098791, "cf_name": "default", "job": 1068, "event": "table_file_creation", "file_number": 10730, "file_size": 1815052, "table_properties": {"data_size": 1799977, "index_size": 7809, "filter_size": 6346, "raw_key_size": 10530, "raw_average_key_size": 21, "raw_value_size": 1787504, "raw_average_value_size": 3582, "num_data_blocks": 245, "num_entries": 499, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 14:53:45.098817 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1068] Level-0 flush table #10730: 1815052 bytes OK
>2018-07-11 14:53:45.148853 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.115387) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10730 started
>2018-07-11 14:53:45.148861 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.148816) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10730: memtable #1 done
>2018-07-11 14:53:45.148863 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.148830) EVENT_LOG_v1 {"time_micros": 1531320825148824, "job": 1068, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:53:45.148869 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.148843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:53:45.148875 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1068] Try to delete WAL files size 3498815, prev total WAL file size 3498815, number of live WAL files 2.
>
>2018-07-11 14:53:45.149738 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:53:45.149918 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 14:53:45.149924 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 82852.0 total, 1000.4 interval
>Cumulative writes: 108K writes, 510K keys, 108K commit groups, 1.0 writes per commit group, ingest: 0.84 GB, 0.01 MB/s
>Cumulative WAL: 108K writes, 108K syncs, 1.00 writes per sync, written: 0.84 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1038 writes, 5139 keys, 1038 commit groups, 1.0 writes per commit group, ingest: 7.43 MB, 0.01 MB/s
>Interval WAL: 1038 writes, 1038 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.73 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        43       535    0.080       0      0
>  L1      1/1   13.10 MB   0.0      8.4     0.7      7.7       7.7     -0.0       0.0  10.8     72.7     66.5       119       532    0.224   1915K   230K
> Sum      2/2   14.83 MB   0.0      8.4     0.7      7.7       8.4      0.7       0.0  11.8     53.5     53.5       162      1067    0.151   1915K   230K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  18.9     54.6     53.3         2        12    0.148     21K   2744
>Uptime(secs): 82852.0 total, 82852.0 interval
>Flush(GB): cumulative 0.717, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.44 GB write, 0.10 MB/s write, 8.44 GB read, 0.10 MB/s read, 161.5 seconds
>Interval compaction: 0.09 GB write, 0.00 MB/s write, 0.09 GB read, 0.00 MB/s read, 1.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.73 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        43       535    0.080       0      0
>  L1      1/1   13.10 MB   0.0      8.4     0.7      7.7       7.7     -0.0       0.0  10.8     72.7     66.5       119       532    0.224   1915K   230K
> Sum      2/2   14.83 MB   0.0      8.4     0.7      7.7       8.4      0.7       0.0  11.8     53.5     53.5       162      1067    0.151   1915K   230K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 82852.0 total, 82852.0 interval
>Flush(GB): cumulative 0.717, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.44 GB write, 0.10 MB/s write, 8.44 GB read, 0.10 MB/s read, 161.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 14:53:45.149944 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.149938) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 14:53:45.149949 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1069] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:53:45.149954 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1068 Base level 0, inputs: [10730(1772KB)], [10728(13MB)]
>
>2018-07-11 14:53:45.149968 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825149957, "job": 1069, "event": "compaction_started", "files_L0": [10730], "files_L1": [10728], "score": -1, "input_data_size": 15547798}
>2018-07-11 14:53:45.332871 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1069] Generated table #10731: 3010 keys, 15406818 bytes
>2018-07-11 14:53:45.332914 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825332888, "cf_name": "default", "job": 1069, "event": "table_file_creation", "file_number": 10731, "file_size": 15406818, "table_properties": {"data_size": 15289289, "index_size": 67252, "filter_size": 49350, "raw_key_size": 61594, "raw_average_key_size": 20, "raw_value_size": 15199084, "raw_average_value_size": 5049, "num_data_blocks": 1967, "num_entries": 3010, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:53:45.367313 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1069] Compacted 1@0 + 1@1 files to L1 => 15406818 bytes
>2018-07-11 14:53:45.409154 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.409085) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.5 rd, 70.9 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.1) out(14.7), read-write-amplify(17.1) write-amplify(8.5) OK, records in: 3286, records dropped: 276
>
>2018-07-11 14:53:45.409163 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:53:45.409106) EVENT_LOG_v1 {"time_micros": 1531320825409097, "job": 1069, "event": "compaction_finished", "compaction_time_micros": 217330, "output_level": 1, "num_output_files": 1, "total_output_size": 15406818, "num_input_records": 3286, "num_output_records": 3010, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:53:45.409588 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825409584, "job": 1069, "event": "table_file_deletion", "file_number": 10730}
>2018-07-11 14:53:45.412315 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320825412312, "job": 1069, "event": "table_file_deletion", "file_number": 10728}
>2018-07-11 14:53:45.412360 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:54:04.172831 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 14:54:10.061727 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10732. Immutable memtables: 0.
>
>2018-07-11 14:54:10.061768 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1070] Syncing log #10729
>2018-07-11 14:54:10.078348 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.061760) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 14:54:10.078359 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1070] Flushing memtable with next log file: 10732
>
>2018-07-11 14:54:10.078372 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850078365, "job": 1070, "event": "flush_started", "num_memtables": 1, "num_entries": 347, "num_deletes": 251, "memory_usage": 213512}
>2018-07-11 14:54:10.078375 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1070] Level-0 flush table #10733: started
>2018-07-11 14:54:10.112191 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850112167, "cf_name": "default", "job": 1070, "event": "table_file_creation", "file_number": 10733, "file_size": 199349, "table_properties": {"data_size": 196258, "index_size": 904, "filter_size": 1268, "raw_key_size": 5742, "raw_average_key_size": 20, "raw_value_size": 191807, "raw_average_value_size": 675, "num_data_blocks": 28, "num_entries": 284, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 14:54:10.112201 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1070] Level-0 flush table #10733: 199349 bytes OK
>2018-07-11 14:54:10.162283 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.128782) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10733 started
>2018-07-11 14:54:10.162291 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.162249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10733: memtable #1 done
>2018-07-11 14:54:10.162294 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.162263) EVENT_LOG_v1 {"time_micros": 1531320850162257, "job": 1070, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 14:54:10.162296 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.162275) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 14:54:10.162301 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1070] Try to delete WAL files size 204318, prev total WAL file size 204318, number of live WAL files 2.
>
>2018-07-11 14:54:10.162418 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 14:54:10.162468 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.162461) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 14:54:10.162477 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1071] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 14:54:10.162483 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1070 Base level 0, inputs: [10733(194KB)], [10731(14MB)]
>
>2018-07-11 14:54:10.162496 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850162486, "job": 1071, "event": "compaction_started", "files_L0": [10733], "files_L1": [10731], "score": -1, "input_data_size": 15606167}
>2018-07-11 14:54:10.337991 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1071] Generated table #10734: 2786 keys, 13635943 bytes
>2018-07-11 14:54:10.338035 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850338008, "cf_name": "default", "job": 1071, "event": "table_file_creation", "file_number": 10734, "file_size": 13635943, "table_properties": {"data_size": 13531206, "index_size": 59912, "filter_size": 43898, "raw_key_size": 57140, "raw_average_key_size": 20, "raw_value_size": 13449263, "raw_average_value_size": 4827, "num_data_blocks": 1743, "num_entries": 2786, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 14:54:10.379497 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1071] Compacted 1@0 + 1@1 files to L1 => 13635943 bytes
>2018-07-11 14:54:10.413049 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.412979) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 71.9 rd, 62.8 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.7) out(13.0), read-write-amplify(146.7) write-amplify(68.4) OK, records in: 3294, records dropped: 508
>
>2018-07-11 14:54:10.413059 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-14:54:10.413001) EVENT_LOG_v1 {"time_micros": 1531320850412991, "job": 1071, "event": "compaction_finished", "compaction_time_micros": 216985, "output_level": 1, "num_output_files": 1, "total_output_size": 13635943, "num_input_records": 3294, "num_output_records": 2786, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 14:54:10.413166 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850413162, "job": 1071, "event": "table_file_deletion", "file_number": 10733}
>2018-07-11 14:54:10.416007 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531320850416004, "job": 1071, "event": "table_file_deletion", "file_number": 10731}
>2018-07-11 14:55:04.173007 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:56:04.173195 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:57:04.173372 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:58:04.173549 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 14:59:04.173732 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:00:04.173918 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:01:04.174171 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:02:04.174359 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:02:05.247581 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10735. Immutable memtables: 0.
>
>2018-07-11 15:02:05.247623 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1072] Syncing log #10732
>2018-07-11 15:02:05.264042 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.247613) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:02:05.264054 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1072] Flushing memtable with next log file: 10735
>
>2018-07-11 15:02:05.264068 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325264060, "job": 1072, "event": "flush_started", "num_memtables": 1, "num_entries": 1914, "num_deletes": 250, "memory_usage": 3551312}
>2018-07-11 15:02:05.264071 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1072] Level-0 flush table #10736: started
>2018-07-11 15:02:05.314551 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325314536, "cf_name": "default", "job": 1072, "event": "table_file_creation", "file_number": 10736, "file_size": 1822510, "table_properties": {"data_size": 1807378, "index_size": 7841, "filter_size": 6371, "raw_key_size": 10550, "raw_average_key_size": 21, "raw_value_size": 1794867, "raw_average_value_size": 3589, "num_data_blocks": 246, "num_entries": 500, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:02:05.314562 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1072] Level-0 flush table #10736: 1822510 bytes OK
>2018-07-11 15:02:05.364606 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.331108) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10736 started
>2018-07-11 15:02:05.364614 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.364568) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10736: memtable #1 done
>2018-07-11 15:02:05.364617 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.364583) EVENT_LOG_v1 {"time_micros": 1531321325364577, "job": 1072, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:02:05.364619 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.364596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:02:05.364625 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1072] Try to delete WAL files size 3513584, prev total WAL file size 3513584, number of live WAL files 2.
>
>2018-07-11 15:02:05.365559 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:02:05.365611 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.365604) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:02:05.365622 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1073] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:02:05.365628 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1072 Base level 0, inputs: [10736(1779KB)], [10734(13MB)]
>
>2018-07-11 15:02:05.365641 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325365631, "job": 1073, "event": "compaction_started", "files_L0": [10736], "files_L1": [10734], "score": -1, "input_data_size": 15458453}
>2018-07-11 15:02:05.548664 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1073] Generated table #10737: 3011 keys, 15324254 bytes
>2018-07-11 15:02:05.548714 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325548687, "cf_name": "default", "job": 1073, "event": "table_file_creation", "file_number": 10737, "file_size": 15324254, "table_properties": {"data_size": 15206843, "index_size": 67285, "filter_size": 49199, "raw_key_size": 61614, "raw_average_key_size": 20, "raw_value_size": 15116601, "raw_average_value_size": 5020, "num_data_blocks": 1968, "num_entries": 3011, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:02:05.581833 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1073] Compacted 1@0 + 1@1 files to L1 => 15324254 bytes
>2018-07-11 15:02:05.615361 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.615293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.5 rd, 70.9 wr, level 1, files in(1, 1) out(1) MB in(1.7, 13.0) out(14.6), read-write-amplify(16.9) write-amplify(8.4) OK, records in: 3286, records dropped: 275
>
>2018-07-11 15:02:05.615370 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:05.615315) EVENT_LOG_v1 {"time_micros": 1531321325615305, "job": 1073, "event": "compaction_finished", "compaction_time_micros": 216175, "output_level": 1, "num_output_files": 1, "total_output_size": 15324254, "num_input_records": 3286, "num_output_records": 3011, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:02:05.615769 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325615765, "job": 1073, "event": "table_file_deletion", "file_number": 10736}
>2018-07-11 15:02:05.618578 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321325618575, "job": 1073, "event": "table_file_deletion", "file_number": 10734}
>2018-07-11 15:02:05.618642 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:02:27.392536 7f29149e4700  1 mon.magna102@0(leader).log v21526 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21526)
>2018-07-11 15:02:27.476307 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10738. Immutable memtables: 0.
>
>2018-07-11 15:02:27.476348 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1074] Syncing log #10735
>2018-07-11 15:02:27.492892 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.476339) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:02:27.492903 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1074] Flushing memtable with next log file: 10738
>
>2018-07-11 15:02:27.492929 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347492921, "job": 1074, "event": "flush_started", "num_memtables": 1, "num_entries": 341, "num_deletes": 251, "memory_usage": 202472}
>2018-07-11 15:02:27.492933 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1074] Level-0 flush table #10739: started
>2018-07-11 15:02:27.526760 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347526747, "cf_name": "default", "job": 1074, "event": "table_file_creation", "file_number": 10739, "file_size": 189691, "table_properties": {"data_size": 186702, "index_size": 836, "filter_size": 1234, "raw_key_size": 5770, "raw_average_key_size": 20, "raw_value_size": 182255, "raw_average_value_size": 639, "num_data_blocks": 26, "num_entries": 285, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:02:27.526770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1074] Level-0 flush table #10739: 189691 bytes OK
>2018-07-11 15:02:27.585129 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.543305) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10739 started
>2018-07-11 15:02:27.585137 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.585095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10739: memtable #1 done
>2018-07-11 15:02:27.585140 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.585110) EVENT_LOG_v1 {"time_micros": 1531321347585103, "job": 1074, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:02:27.585142 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.585121) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:02:27.585160 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1074] Try to delete WAL files size 193515, prev total WAL file size 193515, number of live WAL files 2.
>
>2018-07-11 15:02:27.585292 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:02:27.585338 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.585332) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:02:27.585348 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1075] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:02:27.585354 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1074 Base level 0, inputs: [10739(185KB)], [10737(14MB)]
>
>2018-07-11 15:02:27.585367 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347585357, "job": 1075, "event": "compaction_started", "files_L0": [10739], "files_L1": [10737], "score": -1, "input_data_size": 15513945}
>2018-07-11 15:02:27.752504 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1075] Generated table #10740: 2786 keys, 13568415 bytes
>2018-07-11 15:02:27.752552 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347752524, "cf_name": "default", "job": 1075, "event": "table_file_creation", "file_number": 10740, "file_size": 13568415, "table_properties": {"data_size": 13463849, "index_size": 59876, "filter_size": 43763, "raw_key_size": 57137, "raw_average_key_size": 20, "raw_value_size": 13381933, "raw_average_value_size": 4803, "num_data_blocks": 1742, "num_entries": 2786, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:02:27.785710 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1075] Compacted 1@0 + 1@1 files to L1 => 13568415 bytes
>2018-07-11 15:02:27.827575 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.827503) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.4 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.6) out(12.9), read-write-amplify(153.3) write-amplify(71.5) OK, records in: 3296, records dropped: 510
>
>2018-07-11 15:02:27.827584 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:02:27.827525) EVENT_LOG_v1 {"time_micros": 1531321347827515, "job": 1075, "event": "compaction_finished", "compaction_time_micros": 200329, "output_level": 1, "num_output_files": 1, "total_output_size": 13568415, "num_input_records": 3296, "num_output_records": 2786, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:02:27.827707 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347827703, "job": 1075, "event": "table_file_deletion", "file_number": 10739}
>2018-07-11 15:02:27.830559 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321347830556, "job": 1075, "event": "table_file_deletion", "file_number": 10737}
>2018-07-11 15:03:04.198308 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:04:04.220224 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:05:04.250353 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:06:04.270842 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:07:04.301115 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:08:04.322985 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:09:04.353292 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:10:04.375454 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:10:25.460770 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10741. Immutable memtables: 0.
>
>2018-07-11 15:10:25.460813 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1076] Syncing log #10738
>2018-07-11 15:10:25.477129 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.460804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:10:25.477141 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1076] Flushing memtable with next log file: 10741
>
>2018-07-11 15:10:25.477167 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825477159, "job": 1076, "event": "flush_started", "num_memtables": 1, "num_entries": 1928, "num_deletes": 250, "memory_usage": 3599520}
>2018-07-11 15:10:25.477170 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1076] Level-0 flush table #10742: started
>2018-07-11 15:10:25.536047 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825536020, "cf_name": "default", "job": 1076, "event": "table_file_creation", "file_number": 10742, "file_size": 1847763, "table_properties": {"data_size": 1832501, "index_size": 7905, "filter_size": 6437, "raw_key_size": 10590, "raw_average_key_size": 21, "raw_value_size": 1819917, "raw_average_value_size": 3625, "num_data_blocks": 248, "num_entries": 502, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:10:25.536058 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1076] Level-0 flush table #10742: 1847763 bytes OK
>2018-07-11 15:10:25.586050 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.552552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10742 started
>2018-07-11 15:10:25.586058 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.586014) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10742: memtable #1 done
>2018-07-11 15:10:25.586060 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.586028) EVENT_LOG_v1 {"time_micros": 1531321825586022, "job": 1076, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:10:25.586075 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.586040) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:10:25.586081 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1076] Try to delete WAL files size 3563242, prev total WAL file size 3563242, number of live WAL files 2.
>
>2018-07-11 15:10:25.586980 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:10:25.587173 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 15:10:25.587188 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 83852.4 total, 1000.4 interval
>Cumulative writes: 109K writes, 514K keys, 109K commit groups, 1.0 writes per commit group, ingest: 0.85 GB, 0.01 MB/s
>Cumulative WAL: 109K writes, 109K syncs, 1.00 writes per sync, written: 0.85 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1010 writes, 4530 keys, 1010 commit groups, 1.0 writes per commit group, ingest: 7.13 MB, 0.01 MB/s
>Interval WAL: 1010 writes, 1010 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.76 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        43       539    0.079       0      0
>  L1      1/1   12.94 MB   0.0      8.5     0.7      7.8       7.8     -0.0       0.0  10.8     72.6     66.5       120       536    0.224   1928K   231K
> Sum      2/2   14.70 MB   0.0      8.5     0.7      7.8       8.5      0.7       0.0  11.8     53.5     53.5       163      1075    0.151   1928K   231K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  15.3     54.1     54.0         1         8    0.137     13K   1569
>Uptime(secs): 83852.4 total, 83852.4 interval
>Flush(GB): cumulative 0.721, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.50 GB write, 0.10 MB/s write, 8.50 GB read, 0.10 MB/s read, 162.6 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.76 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        43       539    0.079       0      0
>  L1      1/1   12.94 MB   0.0      8.5     0.7      7.8       7.8     -0.0       0.0  10.8     72.6     66.5       120       536    0.224   1928K   231K
> Sum      2/2   14.70 MB   0.0      8.5     0.7      7.8       8.5      0.7       0.0  11.8     53.5     53.5       163      1075    0.151   1928K   231K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 83852.4 total, 83852.4 interval
>Flush(GB): cumulative 0.721, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.50 GB write, 0.10 MB/s write, 8.50 GB read, 0.10 MB/s read, 162.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 15:10:25.587209 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.587203) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:10:25.587214 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1077] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:10:25.587220 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1076 Base level 0, inputs: [10742(1804KB)], [10740(12MB)]
>
>2018-07-11 15:10:25.587233 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825587223, "job": 1077, "event": "compaction_started", "files_L0": [10742], "files_L1": [10740], "score": -1, "input_data_size": 15416178}
>2018-07-11 15:10:25.770061 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1077] Generated table #10743: 3015 keys, 15295464 bytes
>2018-07-11 15:10:25.770110 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825770082, "cf_name": "default", "job": 1077, "event": "table_file_creation", "file_number": 10743, "file_size": 15295464, "table_properties": {"data_size": 15177984, "index_size": 67385, "filter_size": 49168, "raw_key_size": 61695, "raw_average_key_size": 20, "raw_value_size": 15087617, "raw_average_value_size": 5004, "num_data_blocks": 1971, "num_entries": 3015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:10:25.803222 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1077] Compacted 1@0 + 1@1 files to L1 => 15295464 bytes
>2018-07-11 15:10:25.836768 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.836693) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.4 rd, 70.8 wr, level 1, files in(1, 1) out(1) MB in(1.8, 12.9) out(14.6), read-write-amplify(16.6) write-amplify(8.3) OK, records in: 3288, records dropped: 273
>
>2018-07-11 15:10:25.836777 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:25.836715) EVENT_LOG_v1 {"time_micros": 1531321825836705, "job": 1077, "event": "compaction_finished", "compaction_time_micros": 215972, "output_level": 1, "num_output_files": 1, "total_output_size": 15295464, "num_input_records": 3288, "num_output_records": 3015, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:10:25.837242 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825837237, "job": 1077, "event": "table_file_deletion", "file_number": 10742}
>2018-07-11 15:10:25.839917 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321825839914, "job": 1077, "event": "table_file_deletion", "file_number": 10740}
>2018-07-11 15:10:25.840030 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:10:44.497418 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10744. Immutable memtables: 0.
>
>2018-07-11 15:10:44.497458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1078] Syncing log #10741
>2018-07-11 15:10:44.514024 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.497449) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:10:44.514035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1078] Flushing memtable with next log file: 10744
>
>2018-07-11 15:10:44.514061 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844514041, "job": 1078, "event": "flush_started", "num_memtables": 1, "num_entries": 326, "num_deletes": 251, "memory_usage": 169520}
>2018-07-11 15:10:44.514064 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1078] Level-0 flush table #10745: started
>2018-07-11 15:10:44.556204 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844556191, "cf_name": "default", "job": 1078, "event": "table_file_creation", "file_number": 10745, "file_size": 157479, "table_properties": {"data_size": 154720, "index_size": 706, "filter_size": 1134, "raw_key_size": 5616, "raw_average_key_size": 20, "raw_value_size": 150496, "raw_average_value_size": 541, "num_data_blocks": 22, "num_entries": 278, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:10:44.556213 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1078] Level-0 flush table #10745: 157479 bytes OK
>2018-07-11 15:10:44.614575 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.572737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10745 started
>2018-07-11 15:10:44.614582 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.614541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10745: memtable #1 done
>2018-07-11 15:10:44.614585 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.614555) EVENT_LOG_v1 {"time_micros": 1531321844614549, "job": 1078, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:10:44.614587 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.614566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:10:44.614604 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1078] Try to delete WAL files size 160835, prev total WAL file size 160835, number of live WAL files 2.
>
>2018-07-11 15:10:44.614725 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:10:44.614772 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.614765) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:10:44.614781 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1079] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:10:44.614787 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1078 Base level 0, inputs: [10745(153KB)], [10743(14MB)]
>
>2018-07-11 15:10:44.614801 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844614791, "job": 1079, "event": "compaction_started", "files_L0": [10745], "files_L1": [10743], "score": -1, "input_data_size": 15452943}
>2018-07-11 15:10:44.783156 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1079] Generated table #10746: 2785 keys, 13572738 bytes
>2018-07-11 15:10:44.783212 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844783197, "cf_name": "default", "job": 1079, "event": "table_file_creation", "file_number": 10746, "file_size": 13572738, "table_properties": {"data_size": 13468208, "index_size": 59841, "filter_size": 43762, "raw_key_size": 57115, "raw_average_key_size": 20, "raw_value_size": 13386331, "raw_average_value_size": 4806, "num_data_blocks": 1741, "num_entries": 2785, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:10:44.816332 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1079] Compacted 1@0 + 1@1 files to L1 => 13572738 bytes
>2018-07-11 15:10:44.849854 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.849782) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.7 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.6) out(12.9), read-write-amplify(184.3) write-amplify(86.2) OK, records in: 3293, records dropped: 508
>
>2018-07-11 15:10:44.849862 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:10:44.849804) EVENT_LOG_v1 {"time_micros": 1531321844849795, "job": 1079, "event": "compaction_finished", "compaction_time_micros": 201515, "output_level": 1, "num_output_files": 1, "total_output_size": 13572738, "num_input_records": 3293, "num_output_records": 2785, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:10:44.849977 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844849973, "job": 1079, "event": "table_file_deletion", "file_number": 10745}
>2018-07-11 15:10:44.852800 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531321844852797, "job": 1079, "event": "table_file_deletion", "file_number": 10743}
>2018-07-11 15:11:04.409737 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:12:04.431728 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:13:04.460624 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:13:27.597837 7f29149e4700  1 mon.magna102@0(leader).log v21527 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21527)
>2018-07-11 15:14:04.483803 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:15:04.513854 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:15:07.614303 7f29149e4700  1 mon.magna102@0(leader).log v21528 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21528)
>2018-07-11 15:15:17.623463 7f29149e4700  1 mon.magna102@0(leader).log v21529 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21529)
>2018-07-11 15:16:04.536112 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:17:04.566236 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:18:04.588118 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:18:45.683210 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10747. Immutable memtables: 0.
>
>2018-07-11 15:18:45.683254 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1080] Syncing log #10744
>2018-07-11 15:18:45.699503 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.683245) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:18:45.699515 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1080] Flushing memtable with next log file: 10747
>
>2018-07-11 15:18:45.699542 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322325699534, "job": 1080, "event": "flush_started", "num_memtables": 1, "num_entries": 1959, "num_deletes": 250, "memory_usage": 3657392}
>2018-07-11 15:18:45.699545 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1080] Level-0 flush table #10748: started
>2018-07-11 15:18:45.750041 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322325750026, "cf_name": "default", "job": 1080, "event": "table_file_creation", "file_number": 10748, "file_size": 1903262, "table_properties": {"data_size": 1887631, "index_size": 8102, "filter_size": 6609, "raw_key_size": 10844, "raw_average_key_size": 21, "raw_value_size": 1874714, "raw_average_value_size": 3647, "num_data_blocks": 254, "num_entries": 514, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:18:45.750051 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1080] Level-0 flush table #10748: 1903262 bytes OK
>2018-07-11 15:18:45.800058 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.766567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10748 started
>2018-07-11 15:18:45.800066 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.800021) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10748: memtable #1 done
>2018-07-11 15:18:45.800069 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.800035) EVENT_LOG_v1 {"time_micros": 1531322325800029, "job": 1080, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:18:45.800071 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.800048) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:18:45.800090 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1080] Try to delete WAL files size 3619552, prev total WAL file size 3619552, number of live WAL files 2.
>
>2018-07-11 15:18:45.800988 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:18:45.801051 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:45.801044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:18:45.801057 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1081] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:18:45.801064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1080 Base level 0, inputs: [10748(1858KB)], [10746(12MB)]
>
>2018-07-11 15:18:45.801077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322325801067, "job": 1081, "event": "compaction_started", "files_L0": [10748], "files_L1": [10746], "score": -1, "input_data_size": 15476000}
>2018-07-11 15:18:45.984081 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1081] Generated table #10749: 3025 keys, 15361474 bytes
>2018-07-11 15:18:45.984129 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322325984100, "cf_name": "default", "job": 1081, "event": "table_file_creation", "file_number": 10749, "file_size": 15361474, "table_properties": {"data_size": 15243555, "index_size": 67625, "filter_size": 49367, "raw_key_size": 61898, "raw_average_key_size": 20, "raw_value_size": 15152870, "raw_average_value_size": 5009, "num_data_blocks": 1978, "num_entries": 3025, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:18:46.017275 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1081] Compacted 1@0 + 1@1 files to L1 => 15361474 bytes
>2018-07-11 15:18:46.059121 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:46.059054) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.6 rd, 71.1 wr, level 1, files in(1, 1) out(1) MB in(1.8, 12.9) out(14.6), read-write-amplify(16.2) write-amplify(8.1) OK, records in: 3299, records dropped: 274
>
>2018-07-11 15:18:46.059130 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:46.059076) EVENT_LOG_v1 {"time_micros": 1531322326059066, "job": 1081, "event": "compaction_finished", "compaction_time_micros": 216179, "output_level": 1, "num_output_files": 1, "total_output_size": 15361474, "num_input_records": 3299, "num_output_records": 3025, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:18:46.059578 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322326059574, "job": 1081, "event": "table_file_deletion", "file_number": 10748}
>2018-07-11 15:18:46.062278 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322326062262, "job": 1081, "event": "table_file_deletion", "file_number": 10746}
>2018-07-11 15:18:46.062310 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:18:56.720305 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10750. Immutable memtables: 0.
>
>2018-07-11 15:18:56.720346 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1082] Syncing log #10747
>2018-07-11 15:18:56.736899 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.720336) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:18:56.736911 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1082] Flushing memtable with next log file: 10750
>
>2018-07-11 15:18:56.736936 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322336736916, "job": 1082, "event": "flush_started", "num_memtables": 1, "num_entries": 298, "num_deletes": 251, "memory_usage": 109088}
>2018-07-11 15:18:56.736939 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1082] Level-0 flush table #10751: started
>2018-07-11 15:18:56.779061 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322336779049, "cf_name": "default", "job": 1082, "event": "table_file_creation", "file_number": 10751, "file_size": 99725, "table_properties": {"data_size": 97414, "index_size": 442, "filter_size": 950, "raw_key_size": 5448, "raw_average_key_size": 20, "raw_value_size": 93492, "raw_average_value_size": 346, "num_data_blocks": 14, "num_entries": 270, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:18:56.779072 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1082] Level-0 flush table #10751: 99725 bytes OK
>2018-07-11 15:18:56.829120 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.795582) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10751 started
>2018-07-11 15:18:56.829128 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.829086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10751: memtable #1 done
>2018-07-11 15:18:56.829130 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.829101) EVENT_LOG_v1 {"time_micros": 1531322336829094, "job": 1082, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:18:56.829145 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.829111) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:18:56.829151 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1082] Try to delete WAL files size 100935, prev total WAL file size 100935, number of live WAL files 2.
>
>2018-07-11 15:18:56.829267 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:18:56.829312 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:56.829306) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:18:56.829321 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1083] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:18:56.829327 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1082 Base level 0, inputs: [10751(97KB)], [10749(14MB)]
>
>2018-07-11 15:18:56.829339 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322336829330, "job": 1083, "event": "compaction_started", "files_L0": [10751], "files_L1": [10749], "score": -1, "input_data_size": 15461199}
>2018-07-11 15:18:57.008829 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1083] Generated table #10752: 2787 keys, 13578987 bytes
>2018-07-11 15:18:57.008878 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322337008850, "cf_name": "default", "job": 1083, "event": "table_file_creation", "file_number": 10752, "file_size": 13578987, "table_properties": {"data_size": 13474478, "index_size": 59809, "filter_size": 43773, "raw_key_size": 57150, "raw_average_key_size": 20, "raw_value_size": 13392587, "raw_average_value_size": 4805, "num_data_blocks": 1740, "num_entries": 2787, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:18:57.038005 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1083] Compacted 1@0 + 1@1 files to L1 => 13578987 bytes
>2018-07-11 15:18:57.071516 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:57.071447) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.1 rd, 65.1 wr, level 1, files in(1, 1) out(1) MB in(0.1, 14.6) out(12.9), read-write-amplify(291.2) write-amplify(136.2) OK, records in: 3295, records dropped: 508
>
>2018-07-11 15:18:57.071524 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:18:57.071469) EVENT_LOG_v1 {"time_micros": 1531322337071459, "job": 1083, "event": "compaction_finished", "compaction_time_micros": 208650, "output_level": 1, "num_output_files": 1, "total_output_size": 13578987, "num_input_records": 3295, "num_output_records": 2787, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:18:57.071631 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322337071626, "job": 1083, "event": "table_file_deletion", "file_number": 10751}
>2018-07-11 15:18:57.074528 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322337074525, "job": 1083, "event": "table_file_deletion", "file_number": 10749}
>2018-07-11 15:19:04.614020 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:20:04.636362 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:21:04.665442 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:22:04.695753 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:23:04.717526 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:24:04.739314 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:25:04.769520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:26:04.790217 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:26:48.427692 7f29149e4700  1 mon.magna102@0(leader).log v21530 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21530)
>2018-07-11 15:27:04.820468 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:27:05.896095 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10753. Immutable memtables: 0.
>
>2018-07-11 15:27:05.896138 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1084] Syncing log #10750
>2018-07-11 15:27:05.912463 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:05.896128) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:27:05.912476 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1084] Flushing memtable with next log file: 10753
>
>2018-07-11 15:27:05.912501 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322825912493, "job": 1084, "event": "flush_started", "num_memtables": 1, "num_entries": 1978, "num_deletes": 250, "memory_usage": 3676616}
>2018-07-11 15:27:05.912505 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1084] Level-0 flush table #10754: started
>2018-07-11 15:27:05.971412 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322825971398, "cf_name": "default", "job": 1084, "event": "table_file_creation", "file_number": 10754, "file_size": 1895590, "table_properties": {"data_size": 1879979, "index_size": 8096, "filter_size": 6595, "raw_key_size": 10892, "raw_average_key_size": 21, "raw_value_size": 1867012, "raw_average_value_size": 3618, "num_data_blocks": 254, "num_entries": 516, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:27:05.971422 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1084] Level-0 flush table #10754: 1895590 bytes OK
>2018-07-11 15:27:06.029792 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:05.987964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10754 started
>2018-07-11 15:27:06.029800 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.029755) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10754: memtable #1 done
>2018-07-11 15:27:06.029802 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.029770) EVENT_LOG_v1 {"time_micros": 1531322826029763, "job": 1084, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:27:06.029805 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.029782) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:27:06.029811 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1084] Try to delete WAL files size 3638092, prev total WAL file size 3638092, number of live WAL files 2.
>
>2018-07-11 15:27:06.030725 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:27:06.030902 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 15:27:06.030909 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 84852.9 total, 1000.4 interval
>Cumulative writes: 110K writes, 519K keys, 110K commit groups, 1.0 writes per commit group, ingest: 0.86 GB, 0.01 MB/s
>Cumulative WAL: 110K writes, 110K syncs, 1.00 writes per sync, written: 0.86 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1018 writes, 4561 keys, 1018 commit groups, 1.0 writes per commit group, ingest: 7.17 MB, 0.01 MB/s
>Interval WAL: 1018 writes, 1018 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.81 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        43       543    0.079       0      0
>  L1      1/1   12.95 MB   0.0      8.6     0.7      7.8       7.8     -0.0       0.0  10.8     72.6     66.5       121       540    0.223   1942K   233K
> Sum      2/2   14.76 MB   0.0      8.6     0.7      7.8       8.6      0.7       0.0  11.8     53.5     53.5       164      1083    0.151   1942K   233K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  15.3     53.5     53.5         1         8    0.138     13K   1563
>Uptime(secs): 84852.9 total, 84852.9 interval
>Flush(GB): cumulative 0.725, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.56 GB write, 0.10 MB/s write, 8.56 GB read, 0.10 MB/s read, 163.7 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.81 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.2        43       543    0.079       0      0
>  L1      1/1   12.95 MB   0.0      8.6     0.7      7.8       7.8     -0.0       0.0  10.8     72.6     66.5       121       540    0.223   1942K   233K
> Sum      2/2   14.76 MB   0.0      8.6     0.7      7.8       8.6      0.7       0.0  11.8     53.5     53.5       164      1083    0.151   1942K   233K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 84852.9 total, 84852.9 interval
>Flush(GB): cumulative 0.725, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.56 GB write, 0.10 MB/s write, 8.56 GB read, 0.10 MB/s read, 163.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 15:27:06.030929 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.030922) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:27:06.030933 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1085] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:27:06.030938 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1084 Base level 0, inputs: [10754(1851KB)], [10752(12MB)]
>
>2018-07-11 15:27:06.030950 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322826030941, "job": 1085, "event": "compaction_started", "files_L0": [10754], "files_L1": [10752], "score": -1, "input_data_size": 15474577}
>2018-07-11 15:27:06.213845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1085] Generated table #10755: 3031 keys, 15387750 bytes
>2018-07-11 15:27:06.213894 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322826213866, "cf_name": "default", "job": 1085, "event": "table_file_creation", "file_number": 10755, "file_size": 15387750, "table_properties": {"data_size": 15269646, "index_size": 67727, "filter_size": 49450, "raw_key_size": 62015, "raw_average_key_size": 20, "raw_value_size": 15178812, "raw_average_value_size": 5007, "num_data_blocks": 1981, "num_entries": 3031, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:27:06.247015 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1085] Compacted 1@0 + 1@1 files to L1 => 15387750 bytes
>2018-07-11 15:27:06.280540 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.280472) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.6 rd, 71.2 wr, level 1, files in(1, 1) out(1) MB in(1.8, 12.9) out(14.7), read-write-amplify(16.3) write-amplify(8.1) OK, records in: 3303, records dropped: 272
>
>2018-07-11 15:27:06.280549 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:06.280494) EVENT_LOG_v1 {"time_micros": 1531322826280484, "job": 1085, "event": "compaction_finished", "compaction_time_micros": 216049, "output_level": 1, "num_output_files": 1, "total_output_size": 15387750, "num_input_records": 3303, "num_output_records": 3031, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:27:06.280942 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322826280938, "job": 1085, "event": "table_file_deletion", "file_number": 10754}
>2018-07-11 15:27:06.283622 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322826283619, "job": 1085, "event": "table_file_deletion", "file_number": 10752}
>2018-07-11 15:27:06.283676 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:27:06.895987 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:27:07.066124 7f29149e4700  1 mon.magna102@0(leader).log v21531 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21531)
>2018-07-11 15:27:08.949688 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10756. Immutable memtables: 0.
>
>2018-07-11 15:27:08.949754 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1086] Syncing log #10753
>2018-07-11 15:27:08.966350 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:08.949745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:27:08.966363 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1086] Flushing memtable with next log file: 10756
>
>2018-07-11 15:27:08.966377 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322828966370, "job": 1086, "event": "flush_started", "num_memtables": 1, "num_entries": 278, "num_deletes": 251, "memory_usage": 68720}
>2018-07-11 15:27:08.966380 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1086] Level-0 flush table #10757: started
>2018-07-11 15:27:09.008448 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322829008436, "cf_name": "default", "job": 1086, "event": "table_file_creation", "file_number": 10757, "file_size": 61724, "table_properties": {"data_size": 59738, "index_size": 243, "filter_size": 824, "raw_key_size": 5392, "raw_average_key_size": 20, "raw_value_size": 55977, "raw_average_value_size": 209, "num_data_blocks": 8, "num_entries": 267, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:27:09.008458 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1086] Level-0 flush table #10757: 61724 bytes OK
>2018-07-11 15:27:09.058487 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.024995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10757 started
>2018-07-11 15:27:09.058495 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.058453) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10757: memtable #1 done
>2018-07-11 15:27:09.058498 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.058468) EVENT_LOG_v1 {"time_micros": 1531322829058462, "job": 1086, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:27:09.058513 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.058479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:27:09.058518 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1086] Try to delete WAL files size 60891, prev total WAL file size 60891, number of live WAL files 2.
>
>2018-07-11 15:27:09.058627 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:27:09.058672 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.058666) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:27:09.058681 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1087] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:27:09.058687 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1086 Base level 0, inputs: [10757(60KB)], [10755(14MB)]
>
>2018-07-11 15:27:09.058699 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322829058690, "job": 1087, "event": "compaction_started", "files_L0": [10757], "files_L1": [10755], "score": -1, "input_data_size": 15449474}
>2018-07-11 15:27:09.225808 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1087] Generated table #10758: 2788 keys, 13558692 bytes
>2018-07-11 15:27:09.225852 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322829225825, "cf_name": "default", "job": 1087, "event": "table_file_creation", "file_number": 10758, "file_size": 13558692, "table_properties": {"data_size": 13454344, "index_size": 59709, "filter_size": 43712, "raw_key_size": 57160, "raw_average_key_size": 20, "raw_value_size": 13372519, "raw_average_value_size": 4796, "num_data_blocks": 1737, "num_entries": 2788, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:27:09.259014 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1087] Compacted 1@0 + 1@1 files to L1 => 13558692 bytes
>2018-07-11 15:27:09.292528 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.292458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.1 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(0.1, 14.7) out(12.9), read-write-amplify(470.0) write-amplify(219.7) OK, records in: 3298, records dropped: 510
>
>2018-07-11 15:27:09.292537 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:27:09.292480) EVENT_LOG_v1 {"time_micros": 1531322829292471, "job": 1087, "event": "compaction_finished", "compaction_time_micros": 200301, "output_level": 1, "num_output_files": 1, "total_output_size": 13558692, "num_input_records": 3298, "num_output_records": 2788, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:27:09.292640 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322829292636, "job": 1087, "event": "table_file_deletion", "file_number": 10757}
>2018-07-11 15:27:09.295545 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531322829295543, "job": 1087, "event": "table_file_deletion", "file_number": 10755}
>2018-07-11 15:28:04.847754 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:28:24.823795 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:28:24.974487 7f29149e4700  1 mon.magna102@0(leader).log v21532 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21532)
>2018-07-11 15:28:26.824736 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:28:26.999852 7f29149e4700  1 mon.magna102@0(leader).log v21533 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21533)
>2018-07-11 15:28:40.830993 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 15:28:41.000937 7f29149e4700  1 mon.magna102@0(leader).log v21534 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21534)
>2018-07-11 15:28:46.833542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:28:47.009852 7f29149e4700  1 mon.magna102@0(leader).log v21535 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21535)
>2018-07-11 15:29:04.877640 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:30:04.868282 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:30:04.907997 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:30:05.025206 7f29149e4700  1 mon.magna102@0(leader).log v21536 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21536)
>2018-07-11 15:30:10.871091 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:30:11.025720 7f29149e4700  1 mon.magna102@0(leader).log v21537 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21537)
>2018-07-11 15:30:24.877112 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 15:30:25.053033 7f29149e4700  1 mon.magna102@0(leader).log v21538 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21538)
>2018-07-11 15:30:30.920106 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:30:31.053752 7f29149e4700  1 mon.magna102@0(leader).log v21539 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21539)
>2018-07-11 15:31:04.922436 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:31:38.042680 7f29149e4700  1 mon.magna102@0(leader).log v21540 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21540)
>2018-07-11 15:31:44.912789 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:31:45.068607 7f29149e4700  1 mon.magna102@0(leader).log v21541 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21541)
>2018-07-11 15:31:50.915481 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-11 15:31:51.069095 7f29149e4700  1 mon.magna102@0(leader).log v21542 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21542)
>2018-07-11 15:31:55.952219 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:31:56.061280 7f29149e4700  1 mon.magna102@0(leader).log v21543 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21543)
>2018-07-11 15:32:04.921640 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,6,10 (REQUEST_SLOW)
>2018-07-11 15:32:04.961824 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:32:05.078852 7f29149e4700  1 mon.magna102@0(leader).log v21544 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21544)
>2018-07-11 15:32:10.962408 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:32:11.079436 7f29149e4700  1 mon.magna102@0(leader).log v21545 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21545)
>2018-07-11 15:33:00.946859 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:33:01.100912 7f29149e4700  1 mon.magna102@0(leader).log v21546 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21546)
>2018-07-11 15:33:04.984364 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:33:08.950317 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:33:09.101643 7f29149e4700  1 mon.magna102@0(leader).log v21547 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21547)
>2018-07-11 15:33:20.955885 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 4,6,10 (REQUEST_SLOW)
>2018-07-11 15:33:21.110936 7f29149e4700  1 mon.magna102@0(leader).log v21548 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21548)
>2018-07-11 15:33:25.994161 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:33:26.086365 7f29149e4700  1 mon.magna102@0(leader).log v21549 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21549)
>2018-07-11 15:33:35.995162 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:33:36.103982 7f29149e4700  1 mon.magna102@0(leader).log v21550 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21550)
>2018-07-11 15:33:46.967377 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:33:47.131235 7f29149e4700  1 mon.magna102@0(leader).log v21551 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21551)
>2018-07-11 15:33:56.006454 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:33:56.131901 7f29149e4700  1 mon.magna102@0(leader).log v21552 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21552)
>2018-07-11 15:34:05.015813 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8792 MB, avail 861 GB
>2018-07-11 15:34:36.989996 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:34:37.143308 7f29149e4700  1 mon.magna102@0(leader).log v21553 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21553)
>2018-07-11 15:34:42.992559 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:34:43.152215 7f29149e4700  1 mon.magna102@0(leader).log v21554 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21554)
>2018-07-11 15:34:43.219282 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10759. Immutable memtables: 0.
>
>2018-07-11 15:34:43.219326 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1088] Syncing log #10756
>2018-07-11 15:34:43.235814 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.219317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:34:43.235825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1088] Flushing memtable with next log file: 10759
>
>2018-07-11 15:34:43.235839 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283235831, "job": 1088, "event": "flush_started", "num_memtables": 1, "num_entries": 2029, "num_deletes": 251, "memory_usage": 3945888}
>2018-07-11 15:34:43.235842 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1088] Level-0 flush table #10760: started
>2018-07-11 15:34:43.312611 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283312596, "cf_name": "default", "job": 1088, "event": "table_file_creation", "file_number": 10760, "file_size": 3747259, "table_properties": {"data_size": 3717121, "index_size": 16747, "filter_size": 12466, "raw_key_size": 16184, "raw_average_key_size": 20, "raw_value_size": 3694241, "raw_average_value_size": 4718, "num_data_blocks": 503, "num_entries": 783, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:34:43.312635 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1088] Level-0 flush table #10760: 3747259 bytes OK
>2018-07-11 15:34:43.362610 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.329111) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10760 started
>2018-07-11 15:34:43.362618 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.362572) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10760: memtable #1 done
>2018-07-11 15:34:43.362620 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.362586) EVENT_LOG_v1 {"time_micros": 1531323283362580, "job": 1088, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:34:43.362622 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.362599) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:34:43.362629 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1088] Try to delete WAL files size 3907125, prev total WAL file size 3907125, number of live WAL files 2.
>
>2018-07-11 15:34:43.363595 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:34:43.363652 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.363644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:34:43.363660 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1089] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:34:43.363667 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1088 Base level 0, inputs: [10760(3659KB)], [10758(12MB)]
>
>2018-07-11 15:34:43.363680 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283363670, "job": 1089, "event": "compaction_started", "files_L0": [10760], "files_L1": [10758], "score": -1, "input_data_size": 17305951}
>2018-07-11 15:34:43.546644 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1089] Generated table #10761: 3061 keys, 15417115 bytes
>2018-07-11 15:34:43.546692 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283546665, "cf_name": "default", "job": 1089, "event": "table_file_creation", "file_number": 10761, "file_size": 15417115, "table_properties": {"data_size": 15298046, "index_size": 68515, "filter_size": 49627, "raw_key_size": 63097, "raw_average_key_size": 20, "raw_value_size": 15206086, "raw_average_value_size": 4967, "num_data_blocks": 1989, "num_entries": 3061, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:34:43.588143 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1089] Compacted 1@0 + 1@1 files to L1 => 15417115 bytes
>2018-07-11 15:34:43.621682 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.621614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.1 rd, 68.7 wr, level 1, files in(1, 1) out(1) MB in(3.6, 12.9) out(14.7), read-write-amplify(8.7) write-amplify(4.1) OK, records in: 3571, records dropped: 510
>
>2018-07-11 15:34:43.621691 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:34:43.621635) EVENT_LOG_v1 {"time_micros": 1531323283621626, "job": 1089, "event": "compaction_finished", "compaction_time_micros": 224448, "output_level": 1, "num_output_files": 1, "total_output_size": 15417115, "num_input_records": 3571, "num_output_records": 3061, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:34:43.622442 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283622438, "job": 1089, "event": "table_file_deletion", "file_number": 10760}
>2018-07-11 15:34:43.625151 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323283625149, "job": 1089, "event": "table_file_deletion", "file_number": 10758}
>2018-07-11 15:34:58.999903 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:34:59.173355 7f29149e4700  1 mon.magna102@0(leader).log v21555 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21555)
>2018-07-11 15:35:05.040241 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:35:06.040166 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:35:06.157270 7f29149e4700  1 mon.magna102@0(leader).log v21556 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21556)
>2018-07-11 15:35:16.041164 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:35:16.166554 7f29149e4700  1 mon.magna102@0(leader).log v21557 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21557)
>2018-07-11 15:35:21.050082 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:35:21.167176 7f29149e4700  1 mon.magna102@0(leader).log v21558 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21558)
>2018-07-11 15:35:26.117801 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10762. Immutable memtables: 0.
>
>2018-07-11 15:35:26.117843 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1090] Syncing log #10759
>2018-07-11 15:35:26.134345 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.117834) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:35:26.134356 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1090] Flushing memtable with next log file: 10762
>
>2018-07-11 15:35:26.134383 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326134375, "job": 1090, "event": "flush_started", "num_memtables": 1, "num_entries": 434, "num_deletes": 250, "memory_usage": 420336}
>2018-07-11 15:35:26.134386 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1090] Level-0 flush table #10763: started
>2018-07-11 15:35:26.176539 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326176526, "cf_name": "default", "job": 1090, "event": "table_file_creation", "file_number": 10763, "file_size": 293379, "table_properties": {"data_size": 289737, "index_size": 1185, "filter_size": 1538, "raw_key_size": 6525, "raw_average_key_size": 21, "raw_value_size": 284831, "raw_average_value_size": 955, "num_data_blocks": 37, "num_entries": 298, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:35:26.176549 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1090] Level-0 flush table #10763: 293379 bytes OK
>2018-07-11 15:35:26.226560 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.193073) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10763 started
>2018-07-11 15:35:26.226568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.226525) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10763: memtable #1 done
>2018-07-11 15:35:26.226570 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.226539) EVENT_LOG_v1 {"time_micros": 1531323326226533, "job": 1090, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:35:26.226572 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.226550) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:35:26.226590 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1090] Try to delete WAL files size 410142, prev total WAL file size 410142, number of live WAL files 2.
>
>2018-07-11 15:35:26.226754 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:35:26.226804 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.226797) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:35:26.226814 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1091] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:35:26.226820 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1090 Base level 0, inputs: [10763(286KB)], [10761(14MB)]
>
>2018-07-11 15:35:26.226833 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326226823, "job": 1091, "event": "compaction_started", "files_L0": [10763], "files_L1": [10761], "score": -1, "input_data_size": 15710494}
>2018-07-11 15:35:26.402283 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1091] Generated table #10764: 2866 keys, 14093888 bytes
>2018-07-11 15:35:26.402322 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326402306, "cf_name": "default", "job": 1091, "event": "table_file_creation", "file_number": 10764, "file_size": 14093888, "table_properties": {"data_size": 13986155, "index_size": 61525, "filter_size": 45281, "raw_key_size": 58743, "raw_average_key_size": 20, "raw_value_size": 13901953, "raw_average_value_size": 4850, "num_data_blocks": 1790, "num_entries": 2866, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:35:26.435405 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1091] Compacted 1@0 + 1@1 files to L1 => 14093888 bytes
>2018-07-11 15:35:26.470258 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.470177) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.3 rd, 67.6 wr, level 1, files in(1, 1) out(1) MB in(0.3, 14.7) out(13.4), read-write-amplify(101.6) write-amplify(48.0) OK, records in: 3359, records dropped: 493
>
>2018-07-11 15:35:26.470268 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:35:26.470210) EVENT_LOG_v1 {"time_micros": 1531323326470199, "job": 1091, "event": "compaction_finished", "compaction_time_micros": 208556, "output_level": 1, "num_output_files": 1, "total_output_size": 14093888, "num_input_records": 3359, "num_output_records": 2866, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:35:26.470404 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326470399, "job": 1091, "event": "table_file_deletion", "file_number": 10763}
>2018-07-11 15:35:26.473312 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323326473309, "job": 1091, "event": "table_file_deletion", "file_number": 10761}
>2018-07-11 15:35:26.473377 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:35:31.014533 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:35:31.187922 7f29149e4700  1 mon.magna102@0(leader).log v21559 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21559)
>2018-07-11 15:36:05.073323 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:36:11.032382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:36:11.182409 7f29149e4700  1 mon.magna102@0(leader).log v21560 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21560)
>2018-07-11 15:36:19.035790 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:36:19.191595 7f29149e4700  1 mon.magna102@0(leader).log v21561 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21561)
>2018-07-11 15:36:31.041338 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 4,6,10 (REQUEST_SLOW)
>2018-07-11 15:36:31.193808 7f29149e4700  1 mon.magna102@0(leader).log v21562 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21562)
>2018-07-11 15:36:39.044667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:36:39.194488 7f29149e4700  1 mon.magna102@0(leader).log v21563 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21563)
>2018-07-11 15:36:51.050314 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:36:51.203802 7f29149e4700  1 mon.magna102@0(leader).log v21564 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21564)
>2018-07-11 15:36:57.052986 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:36:57.204331 7f29149e4700  1 mon.magna102@0(leader).log v21565 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21565)
>2018-07-11 15:37:05.096369 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:37:11.059271 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:37:11.213795 7f29149e4700  1 mon.magna102@0(leader).log v21566 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21566)
>2018-07-11 15:37:16.096998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:37:16.205998 7f29149e4700  1 mon.magna102@0(leader).log v21567 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21567)
>2018-07-11 15:37:47.075312 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:37:47.234578 7f29149e4700  1 mon.magna102@0(leader).log v21568 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21568)
>2018-07-11 15:37:53.078064 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:37:53.226818 7f29149e4700  1 mon.magna102@0(leader).log v21569 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21569)
>2018-07-11 15:38:05.110779 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:38:09.085126 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:38:09.236362 7f29149e4700  1 mon.magna102@0(leader).log v21570 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21570)
>2018-07-11 15:38:16.119942 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:38:16.220315 7f29149e4700  1 mon.magna102@0(leader).log v21571 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21571)
>2018-07-11 15:38:35.097125 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:38:35.255233 7f29149e4700  1 mon.magna102@0(leader).log v21572 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21572)
>2018-07-11 15:38:41.099896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:38:41.257090 7f29149e4700  1 mon.magna102@0(leader).log v21573 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21573)
>2018-07-11 15:38:51.104361 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,6,10 (REQUEST_SLOW)
>2018-07-11 15:38:51.274446 7f29149e4700  1 mon.magna102@0(leader).log v21574 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21574)
>2018-07-11 15:38:57.106909 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:38:57.275212 7f29149e4700  1 mon.magna102@0(leader).log v21575 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21575)
>2018-07-11 15:39:05.150561 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:39:21.117786 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:39:21.268672 7f29149e4700  1 mon.magna102@0(leader).log v21576 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21576)
>2018-07-11 15:39:29.121202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:39:29.270636 7f29149e4700  1 mon.magna102@0(leader).log v21577 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21577)
>2018-07-11 15:39:41.126676 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 4,6,10 (REQUEST_SLOW)
>2018-07-11 15:39:41.288309 7f29149e4700  1 mon.magna102@0(leader).log v21578 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21578)
>2018-07-11 15:39:49.130231 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:39:49.272425 7f29149e4700  1 mon.magna102@0(leader).log v21579 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21579)
>2018-07-11 15:40:05.173676 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:40:15.142273 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:40:15.291383 7f29149e4700  1 mon.magna102@0(leader).log v21580 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21580)
>2018-07-11 15:40:21.145014 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:40:21.300303 7f29149e4700  1 mon.magna102@0(leader).log v21581 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21581)
>2018-07-11 15:40:35.151239 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,6,10 (REQUEST_SLOW)
>2018-07-11 15:40:35.319436 7f29149e4700  1 mon.magna102@0(leader).log v21582 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21582)
>2018-07-11 15:40:41.194710 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:40:41.328392 7f29149e4700  1 mon.magna102@0(leader).log v21583 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21583)
>2018-07-11 15:40:57.161122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:40:57.338000 7f29149e4700  1 mon.magna102@0(leader).log v21584 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21584)
>2018-07-11 15:41:03.163742 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:41:03.313600 7f29149e4700  1 mon.magna102@0(leader).log v21585 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21585)
>2018-07-11 15:41:05.197003 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 15:41:21.172045 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 4,6,10 (REQUEST_SLOW)
>2018-07-11 15:41:21.323431 7f29149e4700  1 mon.magna102@0(leader).log v21586 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21586)
>2018-07-11 15:41:26.206595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:41:26.332223 7f29149e4700  1 mon.magna102@0(leader).log v21587 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21587)
>2018-07-11 15:41:55.187411 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:41:55.327676 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10765. Immutable memtables: 0.
>
>2018-07-11 15:41:55.361036 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.327745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:41:55.361049 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1092] Flushing memtable with next log file: 10765
>
>2018-07-11 15:41:55.361063 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715361056, "job": 1092, "event": "flush_started", "num_memtables": 1, "num_entries": 1853, "num_deletes": 251, "memory_usage": 3609744}
>2018-07-11 15:41:55.361066 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1092] Level-0 flush table #10766: started
>2018-07-11 15:41:55.436801 7f29149e4700  1 mon.magna102@0(leader).log v21588 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21588)
>2018-07-11 15:41:55.461596 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715461581, "cf_name": "default", "job": 1092, "event": "table_file_creation", "file_number": 10766, "file_size": 3424910, "table_properties": {"data_size": 3397596, "index_size": 15001, "filter_size": 11389, "raw_key_size": 15206, "raw_average_key_size": 20, "raw_value_size": 3376622, "raw_average_value_size": 4581, "num_data_blocks": 451, "num_entries": 737, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:41:55.461607 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1092] Level-0 flush table #10766: 3424910 bytes OK
>2018-07-11 15:41:55.511681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.478242) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10766 started
>2018-07-11 15:41:55.511689 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.511642) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10766: memtable #1 done
>2018-07-11 15:41:55.511691 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.511657) EVENT_LOG_v1 {"time_micros": 1531323715511650, "job": 1092, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:41:55.511693 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.511670) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:41:55.511700 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1092] Try to delete WAL files size 3573832, prev total WAL file size 3593938, number of live WAL files 2.
>
>2018-07-11 15:41:55.512629 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:41:55.512848 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 15:41:55.512857 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 85742.4 total, 889.5 interval
>Cumulative writes: 111K writes, 524K keys, 111K commit groups, 1.0 writes per commit group, ingest: 0.86 GB, 0.01 MB/s
>Cumulative WAL: 111K writes, 111K syncs, 1.00 writes per sync, written: 0.86 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1014 writes, 4602 keys, 1014 commit groups, 1.0 writes per commit group, ingest: 7.60 MB, 0.01 MB/s
>Interval WAL: 1014 writes, 1014 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.27 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        43       547    0.079       0      0
>  L1      1/1   13.44 MB   0.0      8.6     0.7      7.9       7.9     -0.0       0.0  10.8     72.6     66.5       122       544    0.223   1955K   235K
> Sum      2/2   16.71 MB   0.0      8.6     0.7      7.9       8.6      0.7       0.0  11.8     53.5     53.5       165      1091    0.151   1955K   235K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   8.8     51.8     53.5         1         8    0.147     13K   1785
>Uptime(secs): 85742.4 total, 85742.4 interval
>Flush(GB): cumulative 0.732, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.62 GB write, 0.10 MB/s write, 8.62 GB read, 0.10 MB/s read, 164.9 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.27 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        43       547    0.079       0      0
>  L1      1/1   13.44 MB   0.0      8.6     0.7      7.9       7.9     -0.0       0.0  10.8     72.6     66.5       122       544    0.223   1955K   235K
> Sum      2/2   16.71 MB   0.0      8.6     0.7      7.9       8.6      0.7       0.0  11.8     53.5     53.5       165      1091    0.151   1955K   235K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 85742.4 total, 85742.4 interval
>Flush(GB): cumulative 0.732, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.62 GB write, 0.10 MB/s write, 8.62 GB read, 0.10 MB/s read, 164.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 15:41:55.512879 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.512872) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:41:55.512885 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1093] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:41:55.512890 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1092 Base level 0, inputs: [10766(3344KB)], [10764(13MB)]
>
>2018-07-11 15:41:55.512903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715512893, "job": 1093, "event": "compaction_started", "files_L0": [10766], "files_L1": [10764], "score": -1, "input_data_size": 17518798}
>2018-07-11 15:41:55.704059 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1093] Generated table #10767: 3093 keys, 15640751 bytes
>2018-07-11 15:41:55.704108 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715704081, "cf_name": "default", "job": 1093, "event": "table_file_creation", "file_number": 10767, "file_size": 15640751, "table_properties": {"data_size": 15521105, "index_size": 68577, "filter_size": 50142, "raw_key_size": 63702, "raw_average_key_size": 20, "raw_value_size": 15428661, "raw_average_value_size": 4988, "num_data_blocks": 1990, "num_entries": 3093, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:41:55.737220 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1093] Compacted 1@0 + 1@1 files to L1 => 15640751 bytes
>2018-07-11 15:41:55.770746 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.770678) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.1 rd, 69.7 wr, level 1, files in(1, 1) out(1) MB in(3.3, 13.4) out(14.9), read-write-amplify(9.7) write-amplify(4.6) OK, records in: 3603, records dropped: 510
>
>2018-07-11 15:41:55.770755 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:41:55.770699) EVENT_LOG_v1 {"time_micros": 1531323715770690, "job": 1093, "event": "compaction_finished", "compaction_time_micros": 224298, "output_level": 1, "num_output_files": 1, "total_output_size": 15640751, "num_input_records": 3603, "num_output_records": 3093, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:41:55.771450 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715771446, "job": 1093, "event": "table_file_deletion", "file_number": 10766}
>2018-07-11 15:41:55.774218 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323715774216, "job": 1093, "event": "table_file_deletion", "file_number": 10764}
>2018-07-11 15:42:01.190420 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:42:01.346756 7f29149e4700  1 mon.magna102@0(leader).log v21589 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21589)
>2018-07-11 15:42:05.230191 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:42:17.197477 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:42:17.364695 7f29149e4700  1 mon.magna102@0(leader).log v21590 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21590)
>2018-07-11 15:42:26.241283 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:42:26.358325 7f29149e4700  1 mon.magna102@0(leader).log v21591 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21591)
>2018-07-11 15:42:37.206525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:42:37.359686 7f29149e4700  1 mon.magna102@0(leader).log v21592 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21592)
>2018-07-11 15:42:46.243369 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:42:46.343729 7f29149e4700  1 mon.magna102@0(leader).log v21593 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21593)
>2018-07-11 15:42:53.213618 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:42:53.361170 7f29149e4700  1 mon.magna102@0(leader).log v21594 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21594)
>2018-07-11 15:43:01.253227 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:43:01.370263 7f29149e4700  1 mon.magna102@0(leader).log v21595 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21595)
>2018-07-11 15:43:05.253704 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:43:41.235275 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:43:41.391051 7f29149e4700  1 mon.magna102@0(leader).log v21596 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21596)
>2018-07-11 15:43:46.350026 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10768. Immutable memtables: 0.
>
>2018-07-11 15:43:46.350069 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1094] Syncing log #10765
>2018-07-11 15:43:46.366547 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.350058) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:43:46.366559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1094] Flushing memtable with next log file: 10768
>
>2018-07-11 15:43:46.366585 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826366577, "job": 1094, "event": "flush_started", "num_memtables": 1, "num_entries": 712, "num_deletes": 250, "memory_usage": 1057712}
>2018-07-11 15:43:46.366588 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1094] Level-0 flush table #10769: started
>2018-07-11 15:43:46.408761 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826408747, "cf_name": "default", "job": 1094, "event": "table_file_creation", "file_number": 10769, "file_size": 665868, "table_properties": {"data_size": 659676, "index_size": 2602, "filter_size": 2671, "raw_key_size": 7510, "raw_average_key_size": 21, "raw_value_size": 653063, "raw_average_value_size": 1882, "num_data_blocks": 81, "num_entries": 347, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:43:46.408771 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1094] Level-0 flush table #10769: 665868 bytes OK
>2018-07-11 15:43:46.458780 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.425303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10769 started
>2018-07-11 15:43:46.458788 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.458744) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10769: memtable #1 done
>2018-07-11 15:43:46.458790 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.458759) EVENT_LOG_v1 {"time_micros": 1531323826458752, "job": 1094, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:43:46.458805 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.458770) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:43:46.458811 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1094] Try to delete WAL files size 1042558, prev total WAL file size 1042558, number of live WAL files 2.
>
>2018-07-11 15:43:46.459136 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:43:46.459196 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.459189) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:43:46.459206 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1095] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:43:46.459212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1094 Base level 0, inputs: [10769(650KB)], [10767(14MB)]
>
>2018-07-11 15:43:46.459225 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826459215, "job": 1095, "event": "compaction_started", "files_L0": [10769], "files_L1": [10767], "score": -1, "input_data_size": 16306619}
>2018-07-11 15:43:46.642806 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1095] Generated table #10770: 2981 keys, 14922625 bytes
>2018-07-11 15:43:46.642855 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826642828, "cf_name": "default", "job": 1095, "event": "table_file_creation", "file_number": 10770, "file_size": 14922625, "table_properties": {"data_size": 14809744, "index_size": 64265, "filter_size": 47689, "raw_key_size": 61081, "raw_average_key_size": 20, "raw_value_size": 14721987, "raw_average_value_size": 4938, "num_data_blocks": 1870, "num_entries": 2981, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:43:46.676030 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1095] Compacted 1@0 + 1@1 files to L1 => 14922625 bytes
>2018-07-11 15:43:46.709561 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.709489) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.2 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.9) out(14.2), read-write-amplify(46.9) write-amplify(22.4) OK, records in: 3440, records dropped: 459
>
>2018-07-11 15:43:46.709570 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:43:46.709513) EVENT_LOG_v1 {"time_micros": 1531323826709501, "job": 1095, "event": "compaction_finished", "compaction_time_micros": 216790, "output_level": 1, "num_output_files": 1, "total_output_size": 14922625, "num_input_records": 3440, "num_output_records": 2981, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:43:46.709789 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826709785, "job": 1095, "event": "table_file_deletion", "file_number": 10769}
>2018-07-11 15:43:46.712700 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531323826712697, "job": 1095, "event": "table_file_deletion", "file_number": 10767}
>2018-07-11 15:43:46.712818 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:44:01.244426 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:44:01.419644 7f29149e4700  1 mon.magna102@0(leader).log v21597 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21597)
>2018-07-11 15:44:05.286415 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:44:07.247018 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:44:07.420272 7f29149e4700  1 mon.magna102@0(leader).log v21598 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21598)
>2018-07-11 15:44:13.249611 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:44:13.405430 7f29149e4700  1 mon.magna102@0(leader).log v21599 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21599)
>2018-07-11 15:44:29.256982 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:44:29.406704 7f29149e4700  1 mon.magna102@0(leader).log v21600 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21600)
>2018-07-11 15:44:36.318276 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:44:36.390699 7f29149e4700  1 mon.magna102@0(leader).log v21601 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21601)
>2018-07-11 15:45:05.309338 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:45:15.277722 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:45:15.427047 7f29149e4700  1 mon.magna102@0(leader).log v21602 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21602)
>2018-07-11 15:45:27.283019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:45:27.445966 7f29149e4700  1 mon.magna102@0(leader).log v21603 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21603)
>2018-07-11 15:45:41.289415 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,6 (REQUEST_SLOW)
>2018-07-11 15:45:41.455406 7f29149e4700  1 mon.magna102@0(leader).log v21604 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21604)
>2018-07-11 15:45:53.294554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:45:53.464757 7f29149e4700  1 mon.magna102@0(leader).log v21605 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21605)
>2018-07-11 15:46:05.323683 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:46:07.300863 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 4,6,10 (REQUEST_SLOW)
>2018-07-11 15:46:07.449249 7f29149e4700  1 mon.magna102@0(leader).log v21606 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21606)
>2018-07-11 15:46:16.341265 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 4 slow requests are blocked > 32 sec. Implicated osds 2,4,6,10 (REQUEST_SLOW)
>2018-07-11 15:46:16.441711 7f29149e4700  1 mon.magna102@0(leader).log v21607 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21607)
>2018-07-11 15:47:01.325321 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 2,4,10 (REQUEST_SLOW)
>2018-07-11 15:47:01.479702 7f29149e4700  1 mon.magna102@0(leader).log v21608 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21608)
>2018-07-11 15:47:05.323857 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:47:07.327688 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 2,10 (REQUEST_SLOW)
>2018-07-11 15:47:07.480304 7f29149e4700  1 mon.magna102@0(leader).log v21609 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21609)
>2018-07-11 15:47:16.363896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 2 (REQUEST_SLOW)
>2018-07-11 15:47:16.464334 7f29149e4700  1 mon.magna102@0(leader).log v21610 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21610)
>2018-07-11 15:47:17.531125 7f29149e4700  1 mon.magna102@0(leader).log v21611 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21611)
>2018-07-11 15:47:23.334774 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:47:23.490345 7f29149e4700  1 mon.magna102@0(leader).log v21612 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21612)
>2018-07-11 15:48:05.324033 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:48:45.371666 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 4,6)
>2018-07-11 15:48:45.523748 7f29149e4700  1 mon.magna102@0(leader).log v21613 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21613)
>2018-07-11 15:49:05.324205 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:49:05.380589 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:49:05.542165 7f29149e4700  1 mon.magna102@0(leader).log v21614 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21614)
>2018-07-11 15:49:19.535137 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10771. Immutable memtables: 0.
>
>2018-07-11 15:49:19.535188 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1096] Syncing log #10768
>2018-07-11 15:49:19.551676 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.535172) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:49:19.551688 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1096] Flushing memtable with next log file: 10771
>
>2018-07-11 15:49:19.551714 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159551706, "job": 1096, "event": "flush_started", "num_memtables": 1, "num_entries": 1569, "num_deletes": 251, "memory_usage": 2964576}
>2018-07-11 15:49:19.551717 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1096] Level-0 flush table #10772: started
>2018-07-11 15:49:19.618934 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159618920, "cf_name": "default", "job": 1096, "event": "table_file_creation", "file_number": 10772, "file_size": 2814058, "table_properties": {"data_size": 2791309, "index_size": 12349, "filter_size": 9476, "raw_key_size": 13359, "raw_average_key_size": 20, "raw_value_size": 2773437, "raw_average_value_size": 4279, "num_data_blocks": 373, "num_entries": 648, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:49:19.618945 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1096] Level-0 flush table #10772: 2814058 bytes OK
>2018-07-11 15:49:19.668928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.635418) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10772 started
>2018-07-11 15:49:19.668936 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.668891) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10772: memtable #1 done
>2018-07-11 15:49:19.668939 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.668906) EVENT_LOG_v1 {"time_micros": 1531324159668899, "job": 1096, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:49:19.668954 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.668918) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:49:19.668961 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1096] Try to delete WAL files size 2933453, prev total WAL file size 2933453, number of live WAL files 2.
>
>2018-07-11 15:49:19.669733 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:49:19.669797 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.669790) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:49:19.669804 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1097] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:49:19.669810 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1096 Base level 0, inputs: [10772(2748KB)], [10770(14MB)]
>
>2018-07-11 15:49:19.669823 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159669813, "job": 1097, "event": "compaction_started", "files_L0": [10772], "files_L1": [10770], "score": -1, "input_data_size": 17736683}
>2018-07-11 15:49:19.852968 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1097] Generated table #10773: 3119 keys, 15756607 bytes
>2018-07-11 15:49:19.853011 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159852985, "cf_name": "default", "job": 1097, "event": "table_file_creation", "file_number": 10773, "file_size": 15756607, "table_properties": {"data_size": 15636622, "index_size": 68630, "filter_size": 50428, "raw_key_size": 64193, "raw_average_key_size": 20, "raw_value_size": 15543767, "raw_average_value_size": 4983, "num_data_blocks": 1991, "num_entries": 3119, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:49:19.918606 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1097] Compacted 1@0 + 1@1 files to L1 => 15756607 bytes
>2018-07-11 15:49:19.969796 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.969727) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.3 rd, 63.3 wr, level 1, files in(1, 1) out(1) MB in(2.7, 14.2) out(15.0), read-write-amplify(11.9) write-amplify(5.6) OK, records in: 3629, records dropped: 510
>
>2018-07-11 15:49:19.969805 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:49:19.969748) EVENT_LOG_v1 {"time_micros": 1531324159969739, "job": 1097, "event": "compaction_finished", "compaction_time_micros": 248766, "output_level": 1, "num_output_files": 1, "total_output_size": 15756607, "num_input_records": 3629, "num_output_records": 3119, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:49:19.970449 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159970445, "job": 1097, "event": "table_file_deletion", "file_number": 10772}
>2018-07-11 15:49:19.973393 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324159973390, "job": 1097, "event": "table_file_deletion", "file_number": 10770}
>2018-07-11 15:50:05.324344 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:50:31.419656 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 4,6)
>2018-07-11 15:50:31.585165 7f29149e4700  1 mon.magna102@0(leader).log v21615 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21615)
>2018-07-11 15:50:45.425955 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:50:45.570994 7f29149e4700  1 mon.magna102@0(leader).log v21616 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21616)
>2018-07-11 15:51:05.324517 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:52:05.324706 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:52:05.462053 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:52:05.627879 7f29149e4700  1 mon.magna102@0(leader).log v21617 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21617)
>2018-07-11 15:52:06.569922 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10774. Immutable memtables: 0.
>
>2018-07-11 15:52:06.569965 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1098] Syncing log #10771
>2018-07-11 15:52:06.586464 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.569956) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:52:06.586475 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1098] Flushing memtable with next log file: 10774
>
>2018-07-11 15:52:06.586489 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326586481, "job": 1098, "event": "flush_started", "num_memtables": 1, "num_entries": 860, "num_deletes": 250, "memory_usage": 1355600}
>2018-07-11 15:52:06.586492 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1098] Level-0 flush table #10775: started
>2018-07-11 15:52:06.628681 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326628667, "cf_name": "default", "job": 1098, "event": "table_file_creation", "file_number": 10775, "file_size": 754767, "table_properties": {"data_size": 747781, "index_size": 3095, "filter_size": 2972, "raw_key_size": 7704, "raw_average_key_size": 21, "raw_value_size": 740670, "raw_average_value_size": 2074, "num_data_blocks": 97, "num_entries": 357, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 15:52:06.628691 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1098] Level-0 flush table #10775: 754767 bytes OK
>2018-07-11 15:52:06.678741 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.645246) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10775 started
>2018-07-11 15:52:06.678749 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.678705) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10775: memtable #1 done
>2018-07-11 15:52:06.678752 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.678720) EVENT_LOG_v1 {"time_micros": 1531324326678713, "job": 1098, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:52:06.678754 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.678731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:52:06.678760 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1098] Try to delete WAL files size 1338208, prev total WAL file size 1338208, number of live WAL files 2.
>
>2018-07-11 15:52:06.679104 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:52:06.679295 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 15:52:06.679301 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 86353.5 total, 611.2 interval
>Cumulative writes: 111K writes, 527K keys, 111K commit groups, 1.0 writes per commit group, ingest: 0.87 GB, 0.01 MB/s
>Cumulative WAL: 111K writes, 111K syncs, 1.00 writes per sync, written: 0.87 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 674 writes, 3133 keys, 674 commit groups, 1.0 writes per commit group, ingest: 5.05 MB, 0.01 MB/s
>Interval WAL: 674 writes, 674 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   737.08 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       550    0.079       0      0
>  L1      1/1   15.03 MB   0.0      8.7     0.7      7.9       7.9     -0.0       0.0  10.8     72.6     66.5       122       547    0.223   1966K   236K
> Sum      2/2   15.75 MB   0.0      8.7     0.7      7.9       8.7      0.7       0.0  11.8     53.5     53.5       166      1097    0.151   1966K   236K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.9     55.2     54.1         1         6    0.149     10K   1479
>Uptime(secs): 86353.5 total, 86353.5 interval
>Flush(GB): cumulative 0.736, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.67 GB write, 0.10 MB/s write, 8.67 GB read, 0.10 MB/s read, 165.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   737.08 KB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       550    0.079       0      0
>  L1      1/1   15.03 MB   0.0      8.7     0.7      7.9       7.9     -0.0       0.0  10.8     72.6     66.5       122       547    0.223   1966K   236K
> Sum      2/2   15.75 MB   0.0      8.7     0.7      7.9       8.7      0.7       0.0  11.8     53.5     53.5       166      1097    0.151   1966K   236K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 86353.5 total, 86353.5 interval
>Flush(GB): cumulative 0.736, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.67 GB write, 0.10 MB/s write, 8.67 GB read, 0.10 MB/s read, 165.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 15:52:06.679321 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.679315) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 15:52:06.679326 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1099] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:52:06.679331 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1098 Base level 0, inputs: [10775(737KB)], [10773(15MB)]
>
>2018-07-11 15:52:06.679343 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326679334, "job": 1099, "event": "compaction_started", "files_L0": [10775], "files_L1": [10773], "score": -1, "input_data_size": 16511374}
>2018-07-11 15:52:06.862781 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1099] Generated table #10776: 3045 keys, 15319805 bytes
>2018-07-11 15:52:06.862822 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326862796, "cf_name": "default", "job": 1099, "event": "table_file_creation", "file_number": 10776, "file_size": 15319805, "table_properties": {"data_size": 15204207, "index_size": 65773, "filter_size": 48898, "raw_key_size": 62382, "raw_average_key_size": 20, "raw_value_size": 15114487, "raw_average_value_size": 4963, "num_data_blocks": 1914, "num_entries": 3045, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:52:06.904297 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1099] Compacted 1@0 + 1@1 files to L1 => 15319805 bytes
>2018-07-11 15:52:06.937832 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.937764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.4 rd, 68.1 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.0) out(14.6), read-write-amplify(42.2) write-amplify(20.3) OK, records in: 3476, records dropped: 431
>
>2018-07-11 15:52:06.937841 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:52:06.937786) EVENT_LOG_v1 {"time_micros": 1531324326937776, "job": 1099, "event": "compaction_finished", "compaction_time_micros": 224937, "output_level": 1, "num_output_files": 1, "total_output_size": 15319805, "num_input_records": 3476, "num_output_records": 3045, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:52:06.938045 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326938041, "job": 1099, "event": "table_file_deletion", "file_number": 10775}
>2018-07-11 15:52:06.940909 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324326940906, "job": 1099, "event": "table_file_deletion", "file_number": 10773}
>2018-07-11 15:52:06.940962 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:52:11.464765 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:52:11.638919 7f29149e4700  1 mon.magna102@0(leader).log v21618 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21618)
>2018-07-11 15:52:31.473571 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:52:31.615450 7f29149e4700  1 mon.magna102@0(leader).log v21619 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21619)
>2018-07-11 15:53:05.324838 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:53:51.509584 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:53:51.664105 7f29149e4700  1 mon.magna102@0(leader).log v21620 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21620)
>2018-07-11 15:53:55.511503 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:53:55.665797 7f29149e4700  1 mon.magna102@0(leader).log v21621 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21621)
>2018-07-11 15:54:05.325010 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:54:11.518428 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:54:11.683719 7f29149e4700  1 mon.magna102@0(leader).log v21622 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21622)
>2018-07-11 15:55:05.325206 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:55:35.556316 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 4,6)
>2018-07-11 15:55:35.715946 7f29149e4700  1 mon.magna102@0(leader).log v21623 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21623)
>2018-07-11 15:55:55.565255 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:55:55.750793 7f29149e4700  1 mon.magna102@0(leader).log v21624 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21624)
>2018-07-11 15:56:05.325389 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:57:05.325574 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 15:57:15.601553 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:57:15.757700 7f29149e4700  1 mon.magna102@0(leader).log v21625 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21625)
>2018-07-11 15:57:15.824781 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10777. Immutable memtables: 0.
>
>2018-07-11 15:57:15.824825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1100] Syncing log #10774
>2018-07-11 15:57:15.841339 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.824815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 15:57:15.841350 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1100] Flushing memtable with next log file: 10777
>
>2018-07-11 15:57:15.841377 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324635841369, "job": 1100, "event": "flush_started", "num_memtables": 1, "num_entries": 1405, "num_deletes": 251, "memory_usage": 2557064}
>2018-07-11 15:57:15.841380 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1100] Level-0 flush table #10778: started
>2018-07-11 15:57:15.900241 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324635900226, "cf_name": "default", "job": 1100, "event": "table_file_creation", "file_number": 10778, "file_size": 2426538, "table_properties": {"data_size": 2406497, "index_size": 10816, "filter_size": 8301, "raw_key_size": 12245, "raw_average_key_size": 20, "raw_value_size": 2390478, "raw_average_value_size": 4024, "num_data_blocks": 328, "num_entries": 594, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 15:57:15.900252 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1100] Level-0 flush table #10778: 2426538 bytes OK
>2018-07-11 15:57:15.958490 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.916738) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10778 started
>2018-07-11 15:57:15.958499 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.958449) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10778: memtable #1 done
>2018-07-11 15:57:15.958502 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.958465) EVENT_LOG_v1 {"time_micros": 1531324635958458, "job": 1100, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 15:57:15.958504 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.958479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 15:57:15.958510 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1100] Try to delete WAL files size 2529240, prev total WAL file size 2529240, number of live WAL files 2.
>
>2018-07-11 15:57:15.959244 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 15:57:15.959302 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:57:15.959294) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 15:57:15.959314 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1101] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 15:57:15.959321 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1100 Base level 0, inputs: [10778(2369KB)], [10776(14MB)]
>
>2018-07-11 15:57:15.959337 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324635959327, "job": 1101, "event": "compaction_started", "files_L0": [10778], "files_L1": [10776], "score": -1, "input_data_size": 17746343}
>2018-07-11 15:57:16.142619 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1101] Generated table #10779: 3129 keys, 15725159 bytes
>2018-07-11 15:57:16.142663 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324636142636, "cf_name": "default", "job": 1101, "event": "table_file_creation", "file_number": 10779, "file_size": 15725159, "table_properties": {"data_size": 15605239, "index_size": 68607, "filter_size": 50386, "raw_key_size": 64380, "raw_average_key_size": 20, "raw_value_size": 15512268, "raw_average_value_size": 4957, "num_data_blocks": 1990, "num_entries": 3129, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 15:57:16.175790 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1101] Compacted 1@0 + 1@1 files to L1 => 15725159 bytes
>2018-07-11 15:57:16.217591 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:57:16.217511) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.0 rd, 72.7 wr, level 1, files in(1, 1) out(1) MB in(2.3, 14.6) out(15.0), read-write-amplify(13.8) write-amplify(6.5) OK, records in: 3639, records dropped: 510
>
>2018-07-11 15:57:16.217600 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-15:57:16.217545) EVENT_LOG_v1 {"time_micros": 1531324636217535, "job": 1101, "event": "compaction_finished", "compaction_time_micros": 216437, "output_level": 1, "num_output_files": 1, "total_output_size": 15725159, "num_input_records": 3639, "num_output_records": 3129, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 15:57:16.218146 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324636218142, "job": 1101, "event": "table_file_deletion", "file_number": 10778}
>2018-07-11 15:57:16.221197 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324636221194, "job": 1101, "event": "table_file_deletion", "file_number": 10776}
>2018-07-11 15:57:21.604277 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:57:21.753255 7f29149e4700  1 mon.magna102@0(leader).log v21626 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21626)
>2018-07-11 15:57:35.610556 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:57:35.862832 7f29149e4700  1 mon.magna102@0(leader).log v21627 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21627)
>2018-07-11 15:58:05.325763 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:58:55.646678 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 15:58:55.803033 7f29149e4700  1 mon.magna102@0(leader).log v21628 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21628)
>2018-07-11 15:59:01.649349 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 15:59:01.803607 7f29149e4700  1 mon.magna102@0(leader).log v21629 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21629)
>2018-07-11 15:59:05.325951 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 15:59:15.655696 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 4,6 (REQUEST_SLOW)
>2018-07-11 15:59:15.804734 7f29149e4700  1 mon.magna102@0(leader).log v21630 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21630)
>2018-07-11 16:00:05.326127 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:00:26.786251 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10780. Immutable memtables: 0.
>
>2018-07-11 16:00:26.786296 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1102] Syncing log #10777
>2018-07-11 16:00:26.802837 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.786286) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:00:26.802850 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1102] Flushing memtable with next log file: 10780
>
>2018-07-11 16:00:26.802865 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324826802857, "job": 1102, "event": "flush_started", "num_memtables": 1, "num_entries": 960, "num_deletes": 250, "memory_usage": 1578400}
>2018-07-11 16:00:26.802868 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1102] Level-0 flush table #10781: started
>2018-07-11 16:00:26.853377 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324826853363, "cf_name": "default", "job": 1102, "event": "table_file_creation", "file_number": 10781, "file_size": 885898, "table_properties": {"data_size": 877996, "index_size": 3610, "filter_size": 3373, "raw_key_size": 8066, "raw_average_key_size": 21, "raw_value_size": 870263, "raw_average_value_size": 2320, "num_data_blocks": 113, "num_entries": 375, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:00:26.853388 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1102] Level-0 flush table #10781: 885898 bytes OK
>2018-07-11 16:00:26.903368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.869857) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10781 started
>2018-07-11 16:00:26.903377 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.903332) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10781: memtable #1 done
>2018-07-11 16:00:26.903379 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.903347) EVENT_LOG_v1 {"time_micros": 1531324826903340, "job": 1102, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:00:26.903381 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.903358) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:00:26.903400 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1102] Try to delete WAL files size 1558730, prev total WAL file size 1558730, number of live WAL files 2.
>
>2018-07-11 16:00:26.903875 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:00:26.903937 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:00:26.903930) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:00:26.903944 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1103] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:00:26.903950 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1102 Base level 0, inputs: [10781(865KB)], [10779(14MB)]
>
>2018-07-11 16:00:26.903964 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324826903955, "job": 1103, "event": "compaction_started", "files_L0": [10781], "files_L1": [10779], "score": -1, "input_data_size": 16611057}
>2018-07-11 16:00:27.095735 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1103] Generated table #10782: 3085 keys, 15502063 bytes
>2018-07-11 16:00:27.095784 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324827095756, "cf_name": "default", "job": 1103, "event": "table_file_creation", "file_number": 10782, "file_size": 15502063, "table_properties": {"data_size": 15384870, "index_size": 66730, "filter_size": 49536, "raw_key_size": 63195, "raw_average_key_size": 20, "raw_value_size": 15293905, "raw_average_value_size": 4957, "num_data_blocks": 1942, "num_entries": 3085, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:00:27.137250 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1103] Compacted 1@0 + 1@1 files to L1 => 15502063 bytes
>2018-07-11 16:00:27.170788 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:00:27.170720) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.2 rd, 66.5 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.0) out(14.8), read-write-amplify(36.2) write-amplify(17.5) OK, records in: 3504, records dropped: 419
>
>2018-07-11 16:00:27.170797 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:00:27.170742) EVENT_LOG_v1 {"time_micros": 1531324827170732, "job": 1103, "event": "compaction_finished", "compaction_time_micros": 233267, "output_level": 1, "num_output_files": 1, "total_output_size": 15502063, "num_input_records": 3504, "num_output_records": 3085, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:00:27.171054 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324827171050, "job": 1103, "event": "table_file_deletion", "file_number": 10781}
>2018-07-11 16:00:27.173937 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531324827173934, "job": 1103, "event": "table_file_deletion", "file_number": 10779}
>2018-07-11 16:00:27.174050 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:00:41.694394 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 16:00:41.840815 7f29149e4700  1 mon.magna102@0(leader).log v21631 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21631)
>2018-07-11 16:00:45.696293 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 16:00:45.866294 7f29149e4700  1 mon.magna102@0(leader).log v21632 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21632)
>2018-07-11 16:00:47.616365 7f29149e4700  1 mon.magna102@0(leader).log v21633 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21633)
>2018-07-11 16:00:48.750067 7f29149e4700  1 mon.magna102@0(leader).log v21634 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21634)
>2018-07-11 16:00:49.883771 7f29149e4700  1 mon.magna102@0(leader).log v21635 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21635)
>2018-07-11 16:00:50.950613 7f29149e4700  1 mon.magna102@0(leader).log v21636 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21636)
>2018-07-11 16:01:05.326315 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 16:01:07.302152 7f29149e4700  1 mon.magna102@0(leader).log v21637 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21637)
>2018-07-11 16:01:08.368979 7f29149e4700  1 mon.magna102@0(leader).log v21638 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21638)
>2018-07-11 16:01:17.303351 7f29149e4700  1 mon.magna102@0(leader).log v21639 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21639)
>2018-07-11 16:01:23.388507 7f29149e4700  1 mon.magna102@0(leader).log v21640 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21640)
>2018-07-11 16:01:35.718407 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:01:35.889752 7f29149e4700  1 mon.magna102@0(leader).log v21641 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21641)
>2018-07-11 16:01:41.721069 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:01:41.898694 7f29149e4700  1 mon.magna102@0(leader).log v21642 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21642)
>2018-07-11 16:02:05.326445 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 16:02:51.752077 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:02:51.913215 7f29149e4700  1 mon.magna102@0(leader).log v21643 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21643)
>2018-07-11 16:02:55.753959 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:02:55.905340 7f29149e4700  1 mon.magna102@0(leader).log v21644 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21644)
>2018-07-11 16:03:05.326629 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 16:03:11.760879 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:03:11.914933 7f29149e4700  1 mon.magna102@0(leader).log v21645 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21645)
>2018-07-11 16:03:16.798129 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:03:16.907037 7f29149e4700  1 mon.magna102@0(leader).log v21646 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21646)
>2018-07-11 16:04:05.326816 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:04:25.794125 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:04:25.946717 7f29149e4700  1 mon.magna102@0(leader).log v21647 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21647)
>2018-07-11 16:04:31.796644 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:04:31.947315 7f29149e4700  1 mon.magna102@0(leader).log v21648 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21648)
>2018-07-11 16:04:47.933340 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10783. Immutable memtables: 0.
>
>2018-07-11 16:04:47.933385 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1104] Syncing log #10780
>2018-07-11 16:04:47.949917 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:04:47.933375) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:04:47.949929 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1104] Flushing memtable with next log file: 10783
>
>2018-07-11 16:04:47.949943 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325087949935, "job": 1104, "event": "flush_started", "num_memtables": 1, "num_entries": 1317, "num_deletes": 251, "memory_usage": 2360272}
>2018-07-11 16:04:47.949946 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1104] Level-0 flush table #10784: started
>2018-07-11 16:04:48.008780 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325088008766, "cf_name": "default", "job": 1104, "event": "table_file_creation", "file_number": 10784, "file_size": 2250563, "table_properties": {"data_size": 2231991, "index_size": 9907, "filter_size": 7741, "raw_key_size": 11847, "raw_average_key_size": 20, "raw_value_size": 2216866, "raw_average_value_size": 3848, "num_data_blocks": 301, "num_entries": 576, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:04:48.008791 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1104] Level-0 flush table #10784: 2250563 bytes OK
>2018-07-11 16:04:48.058887 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.025339) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10784 started
>2018-07-11 16:04:48.058897 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.058813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10784: memtable #1 done
>2018-07-11 16:04:48.058899 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.058828) EVENT_LOG_v1 {"time_micros": 1531325088058822, "job": 1104, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:04:48.058901 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.058840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:04:48.058910 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1104] Try to delete WAL files size 2333836, prev total WAL file size 2333836, number of live WAL files 2.
>
>2018-07-11 16:04:48.059551 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:04:48.059731 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 16:04:48.059737 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 87114.9 total, 761.4 interval
>Cumulative writes: 112K writes, 530K keys, 112K commit groups, 1.0 writes per commit group, ingest: 0.87 GB, 0.01 MB/s
>Cumulative WAL: 112K writes, 112K syncs, 1.00 writes per sync, written: 0.87 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 830 writes, 3682 keys, 830 commit groups, 1.0 writes per commit group, ingest: 6.12 MB, 0.01 MB/s
>Interval WAL: 830 writes, 830 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       553    0.079       0      0
>  L1      1/1   14.78 MB   0.0      8.7     0.7      8.0       8.0     -0.0       0.0  10.8     72.6     66.4       123       550    0.223   1976K   238K
> Sum      2/2   16.93 MB   0.0      8.7     0.7      8.0       8.7      0.7       0.0  11.8     53.5     53.5       167      1103    0.151   1976K   238K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.4     54.4     55.7         1         6    0.149     10K   1360
>Uptime(secs): 87114.9 total, 87114.9 interval
>Flush(GB): cumulative 0.741, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.71 GB write, 0.10 MB/s write, 8.71 GB read, 0.10 MB/s read, 166.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.15 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       553    0.079       0      0
>  L1      1/1   14.78 MB   0.0      8.7     0.7      8.0       8.0     -0.0       0.0  10.8     72.6     66.4       123       550    0.223   1976K   238K
> Sum      2/2   16.93 MB   0.0      8.7     0.7      8.0       8.7      0.7       0.0  11.8     53.5     53.5       167      1103    0.151   1976K   238K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 87114.9 total, 87114.9 interval
>Flush(GB): cumulative 0.741, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.71 GB write, 0.10 MB/s write, 8.71 GB read, 0.10 MB/s read, 166.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 16:04:48.059758 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.059752) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:04:48.059763 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1105] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:04:48.059768 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1104 Base level 0, inputs: [10784(2197KB)], [10782(14MB)]
>
>2018-07-11 16:04:48.059780 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325088059771, "job": 1105, "event": "compaction_started", "files_L0": [10784], "files_L1": [10782], "score": -1, "input_data_size": 17752626}
>2018-07-11 16:04:48.259565 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1105] Generated table #10785: 3151 keys, 15735412 bytes
>2018-07-11 16:04:48.259612 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325088259585, "cf_name": "default", "job": 1105, "event": "table_file_creation", "file_number": 10785, "file_size": 15735412, "table_properties": {"data_size": 15615404, "index_size": 68624, "filter_size": 50457, "raw_key_size": 64795, "raw_average_key_size": 20, "raw_value_size": 15522106, "raw_average_value_size": 4926, "num_data_blocks": 1990, "num_entries": 3151, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:04:48.292742 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1105] Compacted 1@0 + 1@1 files to L1 => 15735412 bytes
>2018-07-11 16:04:48.326283 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.326210) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.2 rd, 67.5 wr, level 1, files in(1, 1) out(1) MB in(2.1, 14.8) out(15.0), read-write-amplify(14.9) write-amplify(7.0) OK, records in: 3661, records dropped: 510
>
>2018-07-11 16:04:48.326292 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:04:48.326232) EVENT_LOG_v1 {"time_micros": 1531325088326222, "job": 1105, "event": "compaction_finished", "compaction_time_micros": 232945, "output_level": 1, "num_output_files": 1, "total_output_size": 15735412, "num_input_records": 3661, "num_output_records": 3151, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:04:48.326760 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325088326755, "job": 1105, "event": "table_file_deletion", "file_number": 10784}
>2018-07-11 16:04:48.329802 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325088329799, "job": 1105, "event": "table_file_deletion", "file_number": 10782}
>2018-07-11 16:04:51.805652 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:04:51.952159 7f29149e4700  1 mon.magna102@0(leader).log v21649 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21649)
>2018-07-11 16:04:56.840083 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:04:56.944290 7f29149e4700  1 mon.magna102@0(leader).log v21650 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21650)
>2018-07-11 16:05:05.327006 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:06:05.327122 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:06:05.838885 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:06:05.992302 7f29149e4700  1 mon.magna102@0(leader).log v21651 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21651)
>2018-07-11 16:06:11.841441 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:06:11.992859 7f29149e4700  1 mon.magna102@0(leader).log v21652 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21652)
>2018-07-11 16:06:31.850218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:06:32.019399 7f29149e4700  1 mon.magna102@0(leader).log v21653 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21653)
>2018-07-11 16:07:05.327331 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:07:41.881339 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:07:42.033940 7f29149e4700  1 mon.magna102@0(leader).log v21654 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21654)
>2018-07-11 16:07:45.883223 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:07:46.034399 7f29149e4700  1 mon.magna102@0(leader).log v21655 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21655)
>2018-07-11 16:08:05.327455 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:08:11.894479 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:08:12.054241 7f29149e4700  1 mon.magna102@0(leader).log v21656 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21656)
>2018-07-11 16:08:47.023663 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10786. Immutable memtables: 0.
>
>2018-07-11 16:08:47.023707 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1106] Syncing log #10783
>2018-07-11 16:08:47.040177 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.023697) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:08:47.040195 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1106] Flushing memtable with next log file: 10786
>
>2018-07-11 16:08:47.040208 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327040201, "job": 1106, "event": "flush_started", "num_memtables": 1, "num_entries": 1152, "num_deletes": 250, "memory_usage": 1990584}
>2018-07-11 16:08:47.040211 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1106] Level-0 flush table #10787: started
>2018-07-11 16:08:47.090720 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327090706, "cf_name": "default", "job": 1106, "event": "table_file_creation", "file_number": 10787, "file_size": 1114967, "table_properties": {"data_size": 1105377, "index_size": 4576, "filter_size": 4094, "raw_key_size": 8729, "raw_average_key_size": 21, "raw_value_size": 1096484, "raw_average_value_size": 2687, "num_data_blocks": 143, "num_entries": 408, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:08:47.090730 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1106] Level-0 flush table #10787: 1114967 bytes OK
>2018-07-11 16:08:47.140781 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.107268) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10787 started
>2018-07-11 16:08:47.140789 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.140745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10787: memtable #1 done
>2018-07-11 16:08:47.140791 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.140759) EVENT_LOG_v1 {"time_micros": 1531325327140753, "job": 1106, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:08:47.140794 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.140771) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:08:47.140800 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1106] Try to delete WAL files size 1967624, prev total WAL file size 1967624, number of live WAL files 2.
>
>2018-07-11 16:08:47.141316 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:08:47.141368 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.141362) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:08:47.141377 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1107] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:08:47.141383 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1106 Base level 0, inputs: [10787(1088KB)], [10785(15MB)]
>
>2018-07-11 16:08:47.141396 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327141387, "job": 1107, "event": "compaction_started", "files_L0": [10787], "files_L1": [10785], "score": -1, "input_data_size": 16850379}
>2018-07-11 16:08:47.333146 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1107] Generated table #10788: 3164 keys, 15907440 bytes
>2018-07-11 16:08:47.333208 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327333167, "cf_name": "default", "job": 1107, "event": "table_file_creation", "file_number": 10788, "file_size": 15907440, "table_properties": {"data_size": 15787073, "index_size": 68581, "filter_size": 50859, "raw_key_size": 64801, "raw_average_key_size": 20, "raw_value_size": 15693701, "raw_average_value_size": 4960, "num_data_blocks": 1996, "num_entries": 3164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:08:47.366348 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1107] Compacted 1@0 + 1@1 files to L1 => 15907440 bytes
>2018-07-11 16:08:47.399860 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.399794) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.9 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(1.1, 15.0) out(15.2), read-write-amplify(29.4) write-amplify(14.3) OK, records in: 3559, records dropped: 395
>
>2018-07-11 16:08:47.399869 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:08:47.399816) EVENT_LOG_v1 {"time_micros": 1531325327399806, "job": 1107, "event": "compaction_finished", "compaction_time_micros": 224936, "output_level": 1, "num_output_files": 1, "total_output_size": 15907440, "num_input_records": 3559, "num_output_records": 3164, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:08:47.400139 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327400135, "job": 1107, "event": "table_file_deletion", "file_number": 10787}
>2018-07-11 16:08:47.403025 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325327403022, "job": 1107, "event": "table_file_deletion", "file_number": 10785}
>2018-07-11 16:08:47.403077 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:09:05.327619 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:09:21.926032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:09:22.079427 7f29149e4700  1 mon.magna102@0(leader).log v21657 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21657)
>2018-07-11 16:09:25.927844 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:09:26.079836 7f29149e4700  1 mon.magna102@0(leader).log v21658 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21658)
>2018-07-11 16:09:45.937713 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:09:46.090569 7f29149e4700  1 mon.magna102@0(leader).log v21659 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21659)
>2018-07-11 16:10:05.327793 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:10:55.969051 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:10:56.122228 7f29149e4700  1 mon.magna102@0(leader).log v21660 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21660)
>2018-07-11 16:11:05.327974 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:11:05.973483 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:11:06.124300 7f29149e4700  1 mon.magna102@0(leader).log v21661 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21661)
>2018-07-11 16:11:25.982286 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:11:26.150837 7f29149e4700  1 mon.magna102@0(leader).log v21662 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21662)
>2018-07-11 16:12:05.328153 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:12:36.013588 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:12:36.165479 7f29149e4700  1 mon.magna102@0(leader).log v21663 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21663)
>2018-07-11 16:12:36.232730 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10789. Immutable memtables: 0.
>
>2018-07-11 16:12:36.232774 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1108] Syncing log #10786
>2018-07-11 16:12:36.249110 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.232764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:12:36.249121 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1108] Flushing memtable with next log file: 10789
>
>2018-07-11 16:12:36.249148 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556249140, "job": 1108, "event": "flush_started", "num_memtables": 1, "num_entries": 1117, "num_deletes": 251, "memory_usage": 1912776}
>2018-07-11 16:12:36.249151 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1108] Level-0 flush table #10790: started
>2018-07-11 16:12:36.308001 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556307987, "cf_name": "default", "job": 1108, "event": "table_file_creation", "file_number": 10790, "file_size": 1813479, "table_properties": {"data_size": 1798108, "index_size": 8078, "filter_size": 6373, "raw_key_size": 10504, "raw_average_key_size": 20, "raw_value_size": 1785219, "raw_average_value_size": 3493, "num_data_blocks": 246, "num_entries": 511, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:12:36.308011 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1108] Level-0 flush table #10790: 1813479 bytes OK
>2018-07-11 16:12:36.358004 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.324515) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10790 started
>2018-07-11 16:12:36.358012 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.357967) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10790: memtable #1 done
>2018-07-11 16:12:36.358014 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.357982) EVENT_LOG_v1 {"time_micros": 1531325556357976, "job": 1108, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:12:36.358016 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.357994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:12:36.358035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1108] Try to delete WAL files size 1889468, prev total WAL file size 1889468, number of live WAL files 2.
>
>2018-07-11 16:12:36.358607 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:12:36.358669 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.358662) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:12:36.358676 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1109] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:12:36.358682 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1108 Base level 0, inputs: [10790(1770KB)], [10788(15MB)]
>
>2018-07-11 16:12:36.358695 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556358685, "job": 1109, "event": "compaction_started", "files_L0": [10790], "files_L1": [10788], "score": -1, "input_data_size": 17720919}
>2018-07-11 16:12:36.542047 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1109] Generated table #10791: 3165 keys, 15758274 bytes
>2018-07-11 16:12:36.542097 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556542069, "cf_name": "default", "job": 1109, "event": "table_file_creation", "file_number": 10791, "file_size": 15758274, "table_properties": {"data_size": 15638182, "index_size": 68635, "filter_size": 50530, "raw_key_size": 65058, "raw_average_key_size": 20, "raw_value_size": 15544689, "raw_average_value_size": 4911, "num_data_blocks": 1990, "num_entries": 3165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:12:36.575241 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1109] Compacted 1@0 + 1@1 files to L1 => 15758274 bytes
>2018-07-11 16:12:36.608761 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.608689) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.8 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.2) out(15.0), read-write-amplify(18.5) write-amplify(8.7) OK, records in: 3675, records dropped: 510
>
>2018-07-11 16:12:36.608770 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:12:36.608711) EVENT_LOG_v1 {"time_micros": 1531325556608701, "job": 1109, "event": "compaction_finished", "compaction_time_micros": 216529, "output_level": 1, "num_output_files": 1, "total_output_size": 15758274, "num_input_records": 3675, "num_output_records": 3165, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:12:36.609253 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556609248, "job": 1109, "event": "table_file_deletion", "file_number": 10790}
>2018-07-11 16:12:36.612385 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325556612370, "job": 1109, "event": "table_file_deletion", "file_number": 10788}
>2018-07-11 16:12:42.016352 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:12:42.168163 7f29149e4700  1 mon.magna102@0(leader).log v21664 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21664)
>2018-07-11 16:13:02.025230 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:13:02.178117 7f29149e4700  1 mon.magna102@0(leader).log v21665 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21665)
>2018-07-11 16:13:05.328301 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:13:07.061318 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:13:07.206348 7f29149e4700  1 mon.magna102@0(leader).log v21666 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21666)
>2018-07-11 16:14:05.328472 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:14:16.058212 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:14:16.211269 7f29149e4700  1 mon.magna102@0(leader).log v21667 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21667)
>2018-07-11 16:14:22.060737 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:14:22.211811 7f29149e4700  1 mon.magna102@0(leader).log v21668 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21668)
>2018-07-11 16:14:42.069567 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:14:42.230023 7f29149e4700  1 mon.magna102@0(leader).log v21669 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21669)
>2018-07-11 16:14:47.104880 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:14:47.238813 7f29149e4700  1 mon.magna102@0(leader).log v21670 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21670)
>2018-07-11 16:15:05.328637 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:15:56.102879 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 1,11)
>2018-07-11 16:15:56.253389 7f29149e4700  1 mon.magna102@0(leader).log v21671 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21671)
>2018-07-11 16:16:05.328817 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:16:20.113478 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:16:20.263515 7f29149e4700  1 mon.magna102@0(leader).log v21672 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21672)
>2018-07-11 16:16:27.147032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:16:27.264105 7f29149e4700  1 mon.magna102@0(leader).log v21673 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21673)
>2018-07-11 16:17:05.328982 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:17:07.235148 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10792. Immutable memtables: 0.
>
>2018-07-11 16:17:07.235196 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1110] Syncing log #10789
>2018-07-11 16:17:07.251625 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.235187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:17:07.251637 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1110] Flushing memtable with next log file: 10792
>
>2018-07-11 16:17:07.251650 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827251643, "job": 1110, "event": "flush_started", "num_memtables": 1, "num_entries": 1280, "num_deletes": 250, "memory_usage": 2273168}
>2018-07-11 16:17:07.251653 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1110] Level-0 flush table #10793: started
>2018-07-11 16:17:07.302174 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827302160, "cf_name": "default", "job": 1110, "event": "table_file_creation", "file_number": 10793, "file_size": 1275227, "table_properties": {"data_size": 1264497, "index_size": 5219, "filter_size": 4591, "raw_key_size": 9171, "raw_average_key_size": 21, "raw_value_size": 1254834, "raw_average_value_size": 2918, "num_data_blocks": 163, "num_entries": 430, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:17:07.302195 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1110] Level-0 flush table #10793: 1275227 bytes OK
>2018-07-11 16:17:07.352221 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.318735) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10793 started
>2018-07-11 16:17:07.352229 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.352185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10793: memtable #1 done
>2018-07-11 16:17:07.352231 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.352199) EVENT_LOG_v1 {"time_micros": 1531325827352193, "job": 1110, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:17:07.352233 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.352211) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:17:07.352240 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1110] Try to delete WAL files size 2247688, prev total WAL file size 2247688, number of live WAL files 2.
>
>2018-07-11 16:17:07.352818 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:17:07.352996 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 16:17:07.353002 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 87854.2 total, 739.3 interval
>Cumulative writes: 113K writes, 534K keys, 113K commit groups, 1.0 writes per commit group, ingest: 0.88 GB, 0.01 MB/s
>Cumulative WAL: 113K writes, 113K syncs, 1.00 writes per sync, written: 0.88 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 794 writes, 3549 keys, 794 commit groups, 1.0 writes per commit group, ingest: 5.82 MB, 0.01 MB/s
>Interval WAL: 794 writes, 794 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.22 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       556    0.079       0      0
>  L1      1/1   15.03 MB   0.0      8.8     0.7      8.0       8.0     -0.0       0.0  10.8     72.6     66.5       124       553    0.223   1987K   239K
> Sum      2/2   16.24 MB   0.0      8.8     0.7      8.0       8.8      0.7       0.0  11.8     53.5     53.5       168      1109    0.151   1987K   239K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.3     56.4     55.7         1         6    0.147     10K   1415
>Uptime(secs): 87854.2 total, 87854.2 interval
>Flush(GB): cumulative 0.745, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.76 GB write, 0.10 MB/s write, 8.76 GB read, 0.10 MB/s read, 167.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.22 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       556    0.079       0      0
>  L1      1/1   15.03 MB   0.0      8.8     0.7      8.0       8.0     -0.0       0.0  10.8     72.6     66.5       124       553    0.223   1987K   239K
> Sum      2/2   16.24 MB   0.0      8.8     0.7      8.0       8.8      0.7       0.0  11.8     53.5     53.5       168      1109    0.151   1987K   239K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 87854.2 total, 87854.2 interval
>Flush(GB): cumulative 0.745, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.76 GB write, 0.10 MB/s write, 8.76 GB read, 0.10 MB/s read, 167.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 16:17:07.353023 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.353016) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:17:07.353027 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1111] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:17:07.353033 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1110 Base level 0, inputs: [10793(1245KB)], [10791(15MB)]
>
>2018-07-11 16:17:07.353045 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827353036, "job": 1111, "event": "compaction_started", "files_L0": [10793], "files_L1": [10791], "score": -1, "input_data_size": 17033501}
>2018-07-11 16:17:07.544597 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1111] Generated table #10794: 3216 keys, 16198816 bytes
>2018-07-11 16:17:07.544639 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827544612, "cf_name": "default", "job": 1111, "event": "table_file_creation", "file_number": 10794, "file_size": 16198816, "table_properties": {"data_size": 16076295, "index_size": 69812, "filter_size": 51782, "raw_key_size": 65858, "raw_average_key_size": 20, "raw_value_size": 15981319, "raw_average_value_size": 4969, "num_data_blocks": 2032, "num_entries": 3216, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:17:07.577789 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1111] Compacted 1@0 + 1@1 files to L1 => 16198816 bytes
>2018-07-11 16:17:07.611315 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.611247) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.8 rd, 72.1 wr, level 1, files in(1, 1) out(1) MB in(1.2, 15.0) out(15.4), read-write-amplify(26.1) write-amplify(12.7) OK, records in: 3595, records dropped: 379
>
>2018-07-11 16:17:07.611324 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:17:07.611268) EVENT_LOG_v1 {"time_micros": 1531325827611259, "job": 1111, "event": "compaction_finished", "compaction_time_micros": 224729, "output_level": 1, "num_output_files": 1, "total_output_size": 16198816, "num_input_records": 3595, "num_output_records": 3216, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:17:07.611620 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827611616, "job": 1111, "event": "table_file_deletion", "file_number": 10793}
>2018-07-11 16:17:07.614537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531325827614534, "job": 1111, "event": "table_file_deletion", "file_number": 10791}
>2018-07-11 16:17:07.614587 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:17:36.147651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:17:36.315265 7f29149e4700  1 mon.magna102@0(leader).log v21674 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21674)
>2018-07-11 16:17:42.150207 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:17:42.307583 7f29149e4700  1 mon.magna102@0(leader).log v21675 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21675)
>2018-07-11 16:17:56.156480 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:17:56.308727 7f29149e4700  1 mon.magna102@0(leader).log v21676 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21676)
>2018-07-11 16:18:02.203414 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:18:02.317688 7f29149e4700  1 mon.magna102@0(leader).log v21677 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21677)
>2018-07-11 16:18:05.329160 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:19:05.329354 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:19:16.192302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:19:16.366136 7f29149e4700  1 mon.magna102@0(leader).log v21678 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21678)
>2018-07-11 16:19:22.194798 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:19:22.366715 7f29149e4700  1 mon.magna102@0(leader).log v21679 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21679)
>2018-07-11 16:19:40.203000 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:19:40.368098 7f29149e4700  1 mon.magna102@0(leader).log v21680 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21680)
>2018-07-11 16:19:47.242797 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:19:47.336594 7f29149e4700  1 mon.magna102@0(leader).log v21681 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21681)
>2018-07-11 16:20:05.329524 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:20:16.356427 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10795. Immutable memtables: 0.
>
>2018-07-11 16:20:16.356471 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1112] Syncing log #10792
>2018-07-11 16:20:16.372984 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.356461) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:20:16.372996 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1112] Flushing memtable with next log file: 10795
>
>2018-07-11 16:20:16.373022 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016373014, "job": 1112, "event": "flush_started", "num_memtables": 1, "num_entries": 992, "num_deletes": 251, "memory_usage": 1627048}
>2018-07-11 16:20:16.373026 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1112] Level-0 flush table #10796: started
>2018-07-11 16:20:16.423521 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016423506, "cf_name": "default", "job": 1112, "event": "table_file_creation", "file_number": 10796, "file_size": 1543907, "table_properties": {"data_size": 1530637, "index_size": 6829, "filter_size": 5521, "raw_key_size": 9815, "raw_average_key_size": 20, "raw_value_size": 1519090, "raw_average_value_size": 3178, "num_data_blocks": 208, "num_entries": 478, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:20:16.423531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1112] Level-0 flush table #10796: 1543907 bytes OK
>2018-07-11 16:20:16.481883 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.440054) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10796 started
>2018-07-11 16:20:16.481891 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.481847) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10796: memtable #1 done
>2018-07-11 16:20:16.481894 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.481862) EVENT_LOG_v1 {"time_micros": 1531326016481856, "job": 1112, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:20:16.481896 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.481874) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:20:16.481915 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1112] Try to delete WAL files size 1606268, prev total WAL file size 1606268, number of live WAL files 2.
>
>2018-07-11 16:20:16.482387 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:20:16.482437 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.482430) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:20:16.482446 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1113] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:20:16.482452 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1112 Base level 0, inputs: [10796(1507KB)], [10794(15MB)]
>
>2018-07-11 16:20:16.482466 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016482457, "job": 1113, "event": "compaction_started", "files_L0": [10796], "files_L1": [10794], "score": -1, "input_data_size": 17742723}
>2018-07-11 16:20:16.665920 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1113] Generated table #10797: 3182 keys, 15767261 bytes
>2018-07-11 16:20:16.665970 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016665942, "cf_name": "default", "job": 1113, "event": "table_file_creation", "file_number": 10797, "file_size": 15767261, "table_properties": {"data_size": 15647179, "index_size": 68582, "filter_size": 50573, "raw_key_size": 65372, "raw_average_key_size": 20, "raw_value_size": 15553486, "raw_average_value_size": 4887, "num_data_blocks": 1988, "num_entries": 3182, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:20:16.699109 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1113] Compacted 1@0 + 1@1 files to L1 => 15767261 bytes
>2018-07-11 16:20:16.732644 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.732576) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.9 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(1.5, 15.4) out(15.0), read-write-amplify(21.7) write-amplify(10.2) OK, records in: 3694, records dropped: 512
>
>2018-07-11 16:20:16.732652 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:20:16.732597) EVENT_LOG_v1 {"time_micros": 1531326016732588, "job": 1113, "event": "compaction_finished", "compaction_time_micros": 216626, "output_level": 1, "num_output_files": 1, "total_output_size": 15767261, "num_input_records": 3694, "num_output_records": 3182, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:20:16.733055 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016733052, "job": 1113, "event": "table_file_deletion", "file_number": 10796}
>2018-07-11 16:20:16.736247 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326016736231, "job": 1113, "event": "table_file_deletion", "file_number": 10794}
>2018-07-11 16:20:56.237417 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:20:56.412525 7f29149e4700  1 mon.magna102@0(leader).log v21682 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21682)
>2018-07-11 16:21:02.239922 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:21:02.413105 7f29149e4700  1 mon.magna102@0(leader).log v21683 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21683)
>2018-07-11 16:21:05.329690 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:21:20.247982 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:21:20.422833 7f29149e4700  1 mon.magna102@0(leader).log v21684 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21684)
>2018-07-11 16:21:26.250570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:21:26.448395 7f29149e4700  1 mon.magna102@0(leader).log v21685 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21685)
>2018-07-11 16:22:05.329862 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:22:30.279412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:22:30.429260 7f29149e4700  1 mon.magna102@0(leader).log v21686 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21686)
>2018-07-11 16:22:36.282217 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:22:36.438154 7f29149e4700  1 mon.magna102@0(leader).log v21687 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21687)
>2018-07-11 16:22:56.290989 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:22:56.457585 7f29149e4700  1 mon.magna102@0(leader).log v21688 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21688)
>2018-07-11 16:23:02.293489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:23:02.441525 7f29149e4700  1 mon.magna102@0(leader).log v21689 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21689)
>2018-07-11 16:23:05.330034 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:24:05.330213 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:24:10.324232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:24:10.489383 7f29149e4700  1 mon.magna102@0(leader).log v21690 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21690)
>2018-07-11 16:24:16.326880 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:24:16.481634 7f29149e4700  1 mon.magna102@0(leader).log v21691 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21691)
>2018-07-11 16:24:36.335848 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:24:36.508156 7f29149e4700  1 mon.magna102@0(leader).log v21692 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21692)
>2018-07-11 16:25:05.330395 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:25:27.454922 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10798. Immutable memtables: 0.
>
>2018-07-11 16:25:27.454965 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1114] Syncing log #10795
>2018-07-11 16:25:27.471405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.454955) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:25:27.471416 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1114] Flushing memtable with next log file: 10798
>
>2018-07-11 16:25:27.471443 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327471434, "job": 1114, "event": "flush_started", "num_memtables": 1, "num_entries": 1428, "num_deletes": 250, "memory_usage": 2611536}
>2018-07-11 16:25:27.471446 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1114] Level-0 flush table #10799: started
>2018-07-11 16:25:27.521954 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327521939, "cf_name": "default", "job": 1114, "event": "table_file_creation", "file_number": 10799, "file_size": 1459406, "table_properties": {"data_size": 1447413, "index_size": 5925, "filter_size": 5148, "raw_key_size": 9632, "raw_average_key_size": 21, "raw_value_size": 1436917, "raw_average_value_size": 3172, "num_data_blocks": 185, "num_entries": 453, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:25:27.521964 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1114] Level-0 flush table #10799: 1459406 bytes OK
>2018-07-11 16:25:27.571968 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.538483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10799 started
>2018-07-11 16:25:27.571976 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.571931) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10799: memtable #1 done
>2018-07-11 16:25:27.571978 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.571946) EVENT_LOG_v1 {"time_micros": 1531326327571940, "job": 1114, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:25:27.571993 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.571958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:25:27.572000 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1114] Try to delete WAL files size 2583658, prev total WAL file size 2583658, number of live WAL files 2.
>
>2018-07-11 16:25:27.572690 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:25:27.572752 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.572745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:25:27.572761 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1115] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:25:27.572767 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1114 Base level 0, inputs: [10799(1425KB)], [10797(15MB)]
>
>2018-07-11 16:25:27.572780 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327572770, "job": 1115, "event": "compaction_started", "files_L0": [10799], "files_L1": [10797], "score": -1, "input_data_size": 17226667}
>2018-07-11 16:25:27.772673 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1115] Generated table #10800: 3276 keys, 16528938 bytes
>2018-07-11 16:25:27.772717 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327772690, "cf_name": "default", "job": 1115, "event": "table_file_creation", "file_number": 10800, "file_size": 16528938, "table_properties": {"data_size": 16404001, "index_size": 71185, "filter_size": 52825, "raw_key_size": 67073, "raw_average_key_size": 20, "raw_value_size": 16307215, "raw_average_value_size": 4977, "num_data_blocks": 2072, "num_entries": 3276, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:25:27.814217 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1115] Compacted 1@0 + 1@1 files to L1 => 16528938 bytes
>2018-07-11 16:25:27.847735 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.847663) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.4 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(1.4, 15.0) out(15.8), read-write-amplify(23.1) write-amplify(11.3) OK, records in: 3635, records dropped: 359
>
>2018-07-11 16:25:27.847743 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:25:27.847684) EVENT_LOG_v1 {"time_micros": 1531326327847675, "job": 1115, "event": "compaction_finished", "compaction_time_micros": 241422, "output_level": 1, "num_output_files": 1, "total_output_size": 16528938, "num_input_records": 3635, "num_output_records": 3276, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:25:27.848118 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327848114, "job": 1115, "event": "table_file_deletion", "file_number": 10799}
>2018-07-11 16:25:27.851013 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326327851010, "job": 1115, "event": "table_file_deletion", "file_number": 10797}
>2018-07-11 16:25:27.851127 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:25:50.369546 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:25:50.516846 7f29149e4700  1 mon.magna102@0(leader).log v21693 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21693)
>2018-07-11 16:25:52.370297 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:25:52.517204 7f29149e4700  1 mon.magna102@0(leader).log v21694 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21694)
>2018-07-11 16:26:05.330571 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 16:26:16.381234 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:26:16.536878 7f29149e4700  1 mon.magna102@0(leader).log v21695 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21695)
>2018-07-11 16:27:05.330705 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 16:27:30.414505 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:27:30.560126 7f29149e4700  1 mon.magna102@0(leader).log v21696 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21696)
>2018-07-11 16:27:32.415187 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:27:32.568775 7f29149e4700  1 mon.magna102@0(leader).log v21697 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21697)
>2018-07-11 16:27:52.424132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:27:52.578608 7f29149e4700  1 mon.magna102@0(leader).log v21698 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21698)
>2018-07-11 16:27:57.461853 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:27:57.562328 7f29149e4700  1 mon.magna102@0(leader).log v21699 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21699)
>2018-07-11 16:27:58.721333 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10801. Immutable memtables: 0.
>
>2018-07-11 16:27:58.721376 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1116] Syncing log #10798
>2018-07-11 16:27:58.737876 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.721367) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:27:58.737888 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1116] Flushing memtable with next log file: 10801
>
>2018-07-11 16:27:58.737914 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326478737906, "job": 1116, "event": "flush_started", "num_memtables": 1, "num_entries": 844, "num_deletes": 251, "memory_usage": 1324176}
>2018-07-11 16:27:58.737917 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1116] Level-0 flush table #10802: started
>2018-07-11 16:27:58.780050 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326478780036, "cf_name": "default", "job": 1116, "event": "table_file_creation", "file_number": 10802, "file_size": 1257981, "table_properties": {"data_size": 1246953, "index_size": 5513, "filter_size": 4595, "raw_key_size": 8866, "raw_average_key_size": 20, "raw_value_size": 1237029, "raw_average_value_size": 2856, "num_data_blocks": 168, "num_entries": 433, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:27:58.780076 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1116] Level-0 flush table #10802: 1257981 bytes OK
>2018-07-11 16:27:58.830112 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.796612) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10802 started
>2018-07-11 16:27:58.830120 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.830076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10802: memtable #1 done
>2018-07-11 16:27:58.830123 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.830090) EVENT_LOG_v1 {"time_micros": 1531326478830084, "job": 1116, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:27:58.830125 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.830102) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:27:58.830144 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1116] Try to delete WAL files size 1306021, prev total WAL file size 1306021, number of live WAL files 2.
>
>2018-07-11 16:27:58.830551 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:27:58.830730 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 16:27:58.830737 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 88505.7 total, 651.5 interval
>Cumulative writes: 114K writes, 537K keys, 114K commit groups, 1.0 writes per commit group, ingest: 0.88 GB, 0.01 MB/s
>Cumulative WAL: 114K writes, 114K syncs, 1.00 writes per sync, written: 0.88 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 712 writes, 3264 keys, 712 commit groups, 1.0 writes per commit group, ingest: 5.24 MB, 0.01 MB/s
>Interval WAL: 712 writes, 712 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.20 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       559    0.079       0      0
>  L1      1/1   15.76 MB   0.0      8.8     0.7      8.1       8.1      0.0       0.0  10.8     72.6     66.5       124       556    0.223   1998K   240K
> Sum      2/2   16.96 MB   0.0      8.8     0.7      8.1       8.8      0.7       0.0  11.8     53.6     53.6       168      1115    0.151   1998K   240K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.4     56.6     57.5         1         6    0.146     10K   1250
>Uptime(secs): 88505.7 total, 88505.7 interval
>Flush(GB): cumulative 0.749, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.81 GB write, 0.10 MB/s write, 8.81 GB read, 0.10 MB/s read, 168.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.20 MB   0.0      0.0     0.0      0.0       0.7      0.7       0.0   1.0      0.0     17.3        44       559    0.079       0      0
>  L1      1/1   15.76 MB   0.0      8.8     0.7      8.1       8.1      0.0       0.0  10.8     72.6     66.5       124       556    0.223   1998K   240K
> Sum      2/2   16.96 MB   0.0      8.8     0.7      8.1       8.8      0.7       0.0  11.8     53.6     53.6       168      1115    0.151   1998K   240K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 88505.7 total, 88505.7 interval
>Flush(GB): cumulative 0.749, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.81 GB write, 0.10 MB/s write, 8.81 GB read, 0.10 MB/s read, 168.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 16:27:58.830760 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:27:58.830753) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:27:58.830766 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1117] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:27:58.830771 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1116 Base level 0, inputs: [10802(1228KB)], [10800(15MB)]
>
>2018-07-11 16:27:58.830783 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326478830774, "job": 1117, "event": "compaction_started", "files_L0": [10802], "files_L1": [10800], "score": -1, "input_data_size": 17786919}
>2018-07-11 16:27:59.022503 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1117] Generated table #10803: 3199 keys, 15829346 bytes
>2018-07-11 16:27:59.022546 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326479022520, "cf_name": "default", "job": 1117, "event": "table_file_creation", "file_number": 10803, "file_size": 15829346, "table_properties": {"data_size": 15709093, "index_size": 68596, "filter_size": 50730, "raw_key_size": 65692, "raw_average_key_size": 20, "raw_value_size": 15615151, "raw_average_value_size": 4881, "num_data_blocks": 1988, "num_entries": 3199, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:27:59.055678 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1117] Compacted 1@0 + 1@1 files to L1 => 15829346 bytes
>2018-07-11 16:27:59.089233 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:27:59.089155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 70.4 wr, level 1, files in(1, 1) out(1) MB in(1.2, 15.8) out(15.1), read-write-amplify(26.7) write-amplify(12.6) OK, records in: 3709, records dropped: 510
>
>2018-07-11 16:27:59.089241 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:27:59.089185) EVENT_LOG_v1 {"time_micros": 1531326479089166, "job": 1117, "event": "compaction_finished", "compaction_time_micros": 224879, "output_level": 1, "num_output_files": 1, "total_output_size": 15829346, "num_input_records": 3709, "num_output_records": 3199, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:27:59.089595 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326479089591, "job": 1117, "event": "table_file_deletion", "file_number": 10802}
>2018-07-11 16:27:59.092850 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326479092847, "job": 1117, "event": "table_file_deletion", "file_number": 10800}
>2018-07-11 16:28:05.330890 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:29:05.331080 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:29:10.459874 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:29:10.614217 7f29149e4700  1 mon.magna102@0(leader).log v21700 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21700)
>2018-07-11 16:29:16.462431 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:29:16.614775 7f29149e4700  1 mon.magna102@0(leader).log v21701 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21701)
>2018-07-11 16:29:32.469337 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:29:32.632671 7f29149e4700  1 mon.magna102@0(leader).log v21702 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21702)
>2018-07-11 16:29:37.507557 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:29:37.633145 7f29149e4700  1 mon.magna102@0(leader).log v21703 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21703)
>2018-07-11 16:30:05.331248 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:30:46.502916 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:30:46.656516 7f29149e4700  1 mon.magna102@0(leader).log v21704 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21704)
>2018-07-11 16:30:52.505398 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:30:52.657114 7f29149e4700  1 mon.magna102@0(leader).log v21705 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21705)
>2018-07-11 16:31:05.331421 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:31:10.513579 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:31:10.666825 7f29149e4700  1 mon.magna102@0(leader).log v21706 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21706)
>2018-07-11 16:31:17.550399 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:31:17.650750 7f29149e4700  1 mon.magna102@0(leader).log v21707 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21707)
>2018-07-11 16:32:05.331594 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:32:26.547761 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 1,11)
>2018-07-11 16:32:26.698657 7f29149e4700  1 mon.magna102@0(leader).log v21708 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21708)
>2018-07-11 16:32:46.556504 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 16:32:46.709877 7f29149e4700  1 mon.magna102@0(leader).log v21709 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21709)
>2018-07-11 16:32:52.593432 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:32:52.710414 7f29149e4700  1 mon.magna102@0(leader).log v21710 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21710)
>2018-07-11 16:33:05.331712 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:33:47.690916 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10804. Immutable memtables: 0.
>
>2018-07-11 16:33:47.690960 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1118] Syncing log #10801
>2018-07-11 16:33:47.707339 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.690950) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:33:47.707350 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1118] Flushing memtable with next log file: 10804
>
>2018-07-11 16:33:47.707364 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326827707356, "job": 1118, "event": "flush_started", "num_memtables": 1, "num_entries": 1561, "num_deletes": 250, "memory_usage": 2913280}
>2018-07-11 16:33:47.707367 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1118] Level-0 flush table #10805: started
>2018-07-11 16:33:47.757884 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326827757869, "cf_name": "default", "job": 1118, "event": "table_file_creation", "file_number": 10805, "file_size": 1614189, "table_properties": {"data_size": 1601121, "index_size": 6529, "filter_size": 5619, "raw_key_size": 10012, "raw_average_key_size": 21, "raw_value_size": 1589925, "raw_average_value_size": 3368, "num_data_blocks": 204, "num_entries": 472, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:33:47.757894 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1118] Level-0 flush table #10805: 1614189 bytes OK
>2018-07-11 16:33:47.807886 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.774438) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10805 started
>2018-07-11 16:33:47.807894 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.807850) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10805: memtable #1 done
>2018-07-11 16:33:47.807897 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.807864) EVENT_LOG_v1 {"time_micros": 1531326827807858, "job": 1118, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:33:47.807899 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.807876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:33:47.807905 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1118] Try to delete WAL files size 2882723, prev total WAL file size 2882723, number of live WAL files 2.
>
>2018-07-11 16:33:47.808644 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:33:47.808698 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:33:47.808691) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:33:47.808706 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1119] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:33:47.808713 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1118 Base level 0, inputs: [10805(1576KB)], [10803(15MB)]
>
>2018-07-11 16:33:47.808725 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326827808716, "job": 1119, "event": "compaction_started", "files_L0": [10805], "files_L1": [10803], "score": -1, "input_data_size": 17443535}
>2018-07-11 16:33:48.008586 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1119] Generated table #10806: 3331 keys, 16876140 bytes
>2018-07-11 16:33:48.008636 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326828008608, "cf_name": "default", "job": 1119, "event": "table_file_creation", "file_number": 10806, "file_size": 16876140, "table_properties": {"data_size": 16748850, "index_size": 72483, "filter_size": 53880, "raw_key_size": 68191, "raw_average_key_size": 20, "raw_value_size": 16650370, "raw_average_value_size": 4998, "num_data_blocks": 2110, "num_entries": 3331, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:33:48.041816 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1119] Compacted 1@0 + 1@1 files to L1 => 16876140 bytes
>2018-07-11 16:33:48.075347 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:33:48.075279) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.8 rd, 72.4 wr, level 1, files in(1, 1) out(1) MB in(1.5, 15.1) out(16.1), read-write-amplify(21.3) write-amplify(10.5) OK, records in: 3671, records dropped: 340
>
>2018-07-11 16:33:48.075356 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:33:48.075300) EVENT_LOG_v1 {"time_micros": 1531326828075290, "job": 1119, "event": "compaction_finished", "compaction_time_micros": 233075, "output_level": 1, "num_output_files": 1, "total_output_size": 16876140, "num_input_records": 3671, "num_output_records": 3331, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:33:48.075721 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326828075718, "job": 1119, "event": "table_file_deletion", "file_number": 10805}
>2018-07-11 16:33:48.078635 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326828078632, "job": 1119, "event": "table_file_deletion", "file_number": 10803}
>2018-07-11 16:33:48.078690 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:34:00.590307 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:34:00.743605 7f29149e4700  1 mon.magna102@0(leader).log v21711 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21711)
>2018-07-11 16:34:05.331832 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 16:34:06.592959 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:34:06.744214 7f29149e4700  1 mon.magna102@0(leader).log v21712 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21712)
>2018-07-11 16:34:26.602012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,11 (REQUEST_SLOW)
>2018-07-11 16:34:26.754235 7f29149e4700  1 mon.magna102@0(leader).log v21713 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21713)
>2018-07-11 16:35:00.617310 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:35:00.774633 7f29149e4700  1 mon.magna102@0(leader).log v21714 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21714)
>2018-07-11 16:35:03.783215 7f29149e4700  1 mon.magna102@0(leader).log v21715 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21715)
>2018-07-11 16:35:05.332001 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 16:35:05.941875 7f29149e4700  1 mon.magna102@0(leader).log v21716 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21716)
>2018-07-11 16:35:06.941907 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:35:07.117296 7f29149e4700  1 mon.magna102@0(leader).log v21717 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21717)
>2018-07-11 16:35:25.793623 7f29149e4700  1 mon.magna102@0(leader).log v21718 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21718)
>2018-07-11 16:35:28.360555 7f29149e4700  1 mon.magna102@0(leader).log v21719 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21719)
>2018-07-11 16:35:35.769677 7f29149e4700  1 mon.magna102@0(leader).log v21720 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21720)
>2018-07-11 16:35:35.838061 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10807. Immutable memtables: 0.
>
>2018-07-11 16:35:35.838106 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1120] Syncing log #10804
>2018-07-11 16:35:35.854508 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.838097) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:35:35.854520 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1120] Flushing memtable with next log file: 10807
>
>2018-07-11 16:35:35.854534 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326935854527, "job": 1120, "event": "flush_started", "num_memtables": 1, "num_entries": 714, "num_deletes": 251, "memory_usage": 1040160}
>2018-07-11 16:35:35.854537 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1120] Level-0 flush table #10808: started
>2018-07-11 16:35:35.905116 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326935905102, "cf_name": "default", "job": 1120, "event": "table_file_creation", "file_number": 10808, "file_size": 992635, "table_properties": {"data_size": 983702, "index_size": 4265, "filter_size": 3748, "raw_key_size": 8125, "raw_average_key_size": 20, "raw_value_size": 975177, "raw_average_value_size": 2450, "num_data_blocks": 130, "num_entries": 398, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:35:35.905127 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1120] Level-0 flush table #10808: 992635 bytes OK
>2018-07-11 16:35:35.955176 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.921692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10808 started
>2018-07-11 16:35:35.955191 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.955140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10808: memtable #1 done
>2018-07-11 16:35:35.955193 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.955155) EVENT_LOG_v1 {"time_micros": 1531326935955149, "job": 1120, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:35:35.955195 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.955166) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:35:35.955202 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1120] Try to delete WAL files size 1024434, prev total WAL file size 1024434, number of live WAL files 2.
>
>2018-07-11 16:35:35.955483 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:35:35.955534 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:35:35.955527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:35:35.955542 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1121] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:35:35.955548 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1120 Base level 0, inputs: [10808(969KB)], [10806(16MB)]
>
>2018-07-11 16:35:35.955560 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326935955551, "job": 1121, "event": "compaction_started", "files_L0": [10808], "files_L1": [10806], "score": -1, "input_data_size": 17868775}
>2018-07-11 16:35:36.139252 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1121] Generated table #10809: 3219 keys, 15912763 bytes
>2018-07-11 16:35:36.139284 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326936139269, "cf_name": "default", "job": 1121, "event": "table_file_creation", "file_number": 10809, "file_size": 15912763, "table_properties": {"data_size": 15792249, "index_size": 68642, "filter_size": 50945, "raw_key_size": 66069, "raw_average_key_size": 20, "raw_value_size": 15697993, "raw_average_value_size": 4876, "num_data_blocks": 1989, "num_entries": 3219, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:35:36.172411 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1121] Compacted 1@0 + 1@1 files to L1 => 15912763 bytes
>2018-07-11 16:35:36.205929 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:35:36.205860) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.4 rd, 73.4 wr, level 1, files in(1, 1) out(1) MB in(0.9, 16.1) out(15.2), read-write-amplify(34.0) write-amplify(16.0) OK, records in: 3729, records dropped: 510
>
>2018-07-11 16:35:36.205938 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:35:36.205881) EVENT_LOG_v1 {"time_micros": 1531326936205872, "job": 1121, "event": "compaction_finished", "compaction_time_micros": 216834, "output_level": 1, "num_output_files": 1, "total_output_size": 15912763, "num_input_records": 3729, "num_output_records": 3219, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:35:36.206212 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326936206195, "job": 1121, "event": "table_file_deletion", "file_number": 10808}
>2018-07-11 16:35:36.209586 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531326936209583, "job": 1121, "event": "table_file_deletion", "file_number": 10806}
>2018-07-11 16:35:50.639834 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:35:50.791070 7f29149e4700  1 mon.magna102@0(leader).log v21721 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21721)
>2018-07-11 16:36:05.332177 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:37:00.671277 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:37:00.822367 7f29149e4700  1 mon.magna102@0(leader).log v21722 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21722)
>2018-07-11 16:37:05.332369 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:37:10.675812 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:37:10.823245 7f29149e4700  1 mon.magna102@0(leader).log v21723 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21723)
>2018-07-11 16:37:26.682715 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:37:26.832710 7f29149e4700  1 mon.magna102@0(leader).log v21724 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21724)
>2018-07-11 16:37:32.716423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:37:32.833442 7f29149e4700  1 mon.magna102@0(leader).log v21725 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21725)
>2018-07-11 16:38:05.332535 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:38:46.718902 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:38:46.865006 7f29149e4700  1 mon.magna102@0(leader).log v21726 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21726)
>2018-07-11 16:38:54.722442 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:38:54.875305 7f29149e4700  1 mon.magna102@0(leader).log v21727 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21727)
>2018-07-11 16:39:05.332680 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:39:10.729702 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:39:10.884891 7f29149e4700  1 mon.magna102@0(leader).log v21728 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21728)
>2018-07-11 16:40:05.332818 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:40:26.764096 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:40:26.916705 7f29149e4700  1 mon.magna102@0(leader).log v21729 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21729)
>2018-07-11 16:40:34.767565 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:40:34.967661 7f29149e4700  1 mon.magna102@0(leader).log v21730 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21730)
>2018-07-11 16:40:44.772034 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:40:45.001998 7f29149e4700  1 mon.magna102@0(leader).log v21731 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21731)
>2018-07-11 16:40:50.774899 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:40:50.927667 7f29149e4700  1 mon.magna102@0(leader).log v21732 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21732)
>2018-07-11 16:41:05.333000 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:42:05.333132 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:42:06.808911 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:42:06.984170 7f29149e4700  1 mon.magna102@0(leader).log v21733 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21733)
>2018-07-11 16:42:07.926503 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10810. Immutable memtables: 0.
>
>2018-07-11 16:42:07.926560 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1122] Syncing log #10807
>2018-07-11 16:42:07.942797 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:42:07.926551) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:42:07.942809 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1122] Flushing memtable with next log file: 10810
>
>2018-07-11 16:42:07.942822 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327327942815, "job": 1122, "event": "flush_started", "num_memtables": 1, "num_entries": 1731, "num_deletes": 250, "memory_usage": 3273920}
>2018-07-11 16:42:07.942825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1122] Level-0 flush table #10811: started
>2018-07-11 16:42:08.001674 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327328001659, "cf_name": "default", "job": 1122, "event": "table_file_creation", "file_number": 10811, "file_size": 1805890, "table_properties": {"data_size": 1791374, "index_size": 7366, "filter_size": 6230, "raw_key_size": 10574, "raw_average_key_size": 21, "raw_value_size": 1779182, "raw_average_value_size": 3558, "num_data_blocks": 230, "num_entries": 500, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:42:08.001685 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1122] Level-0 flush table #10811: 1805890 bytes OK
>2018-07-11 16:42:08.052962 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.018232) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10811 started
>2018-07-11 16:42:08.052970 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.052925) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10811: memtable #1 done
>2018-07-11 16:42:08.052972 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.052940) EVENT_LOG_v1 {"time_micros": 1531327328052934, "job": 1122, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:42:08.052975 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.052952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:42:08.052981 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1122] Try to delete WAL files size 3240515, prev total WAL file size 3240515, number of live WAL files 2.
>
>2018-07-11 16:42:08.053785 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:42:08.053964 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 16:42:08.053970 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 89354.9 total, 849.2 interval
>Cumulative writes: 115K writes, 541K keys, 115K commit groups, 1.0 writes per commit group, ingest: 0.89 GB, 0.01 MB/s
>Cumulative WAL: 115K writes, 115K syncs, 1.00 writes per sync, written: 0.89 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 922 writes, 4006 keys, 922 commit groups, 1.0 writes per commit group, ingest: 6.82 MB, 0.01 MB/s
>Interval WAL: 922 writes, 922 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.72 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        44       562    0.079       0      0
>  L1      1/1   15.18 MB   0.0      8.9     0.8      8.1       8.1     -0.0       0.0  10.8     72.6     66.5       125       559    0.223   2009K   242K
> Sum      2/2   16.90 MB   0.0      8.9     0.8      8.1       8.9      0.8       0.0  11.8     53.6     53.6       169      1121    0.151   2009K   242K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.0     57.3     57.2         1         6    0.147     11K   1360
>Uptime(secs): 89354.9 total, 89354.9 interval
>Flush(GB): cumulative 0.753, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.86 GB write, 0.10 MB/s write, 8.86 GB read, 0.10 MB/s read, 169.3 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.72 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        44       562    0.079       0      0
>  L1      1/1   15.18 MB   0.0      8.9     0.8      8.1       8.1     -0.0       0.0  10.8     72.6     66.5       125       559    0.223   2009K   242K
> Sum      2/2   16.90 MB   0.0      8.9     0.8      8.1       8.9      0.8       0.0  11.8     53.6     53.6       169      1121    0.151   2009K   242K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 89354.9 total, 89354.9 interval
>Flush(GB): cumulative 0.753, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.86 GB write, 0.10 MB/s write, 8.86 GB read, 0.10 MB/s read, 169.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 16:42:08.053994 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.053984) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:42:08.054001 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1123] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:42:08.054006 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1122 Base level 0, inputs: [10811(1763KB)], [10809(15MB)]
>
>2018-07-11 16:42:08.054018 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327328054009, "job": 1123, "event": "compaction_started", "files_L0": [10811], "files_L1": [10809], "score": -1, "input_data_size": 17718653}
>2018-07-11 16:42:08.253745 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1123] Generated table #10812: 3401 keys, 17302942 bytes
>2018-07-11 16:42:08.253793 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327328253765, "cf_name": "default", "job": 1123, "event": "table_file_creation", "file_number": 10812, "file_size": 17302942, "table_properties": {"data_size": 17172660, "index_size": 74160, "filter_size": 55195, "raw_key_size": 69614, "raw_average_key_size": 20, "raw_value_size": 17072016, "raw_average_value_size": 5019, "num_data_blocks": 2159, "num_entries": 3401, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:42:08.286911 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1123] Compacted 1@0 + 1@1 files to L1 => 17302942 bytes
>2018-07-11 16:42:08.320395 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.320327) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.1 rd, 74.3 wr, level 1, files in(1, 1) out(1) MB in(1.7, 15.2) out(16.5), read-write-amplify(19.4) write-amplify(9.6) OK, records in: 3719, records dropped: 318
>
>2018-07-11 16:42:08.320404 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:42:08.320349) EVENT_LOG_v1 {"time_micros": 1531327328320339, "job": 1123, "event": "compaction_finished", "compaction_time_micros": 232876, "output_level": 1, "num_output_files": 1, "total_output_size": 17302942, "num_input_records": 3719, "num_output_records": 3401, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:42:08.320807 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327328320803, "job": 1123, "event": "table_file_deletion", "file_number": 10811}
>2018-07-11 16:42:08.323768 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327328323765, "job": 1123, "event": "table_file_deletion", "file_number": 10809}
>2018-07-11 16:42:08.323820 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:42:10.811105 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:42:10.962982 7f29149e4700  1 mon.magna102@0(leader).log v21734 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21734)
>2018-07-11 16:42:30.819999 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:42:31.031195 7f29149e4700  1 mon.magna102@0(leader).log v21735 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21735)
>2018-07-11 16:43:05.333267 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 16:43:22.978152 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10813. Immutable memtables: 0.
>
>2018-07-11 16:43:22.978223 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1124] Syncing log #10810
>2018-07-11 16:43:22.994708 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:43:22.978213) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:43:22.994719 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1124] Flushing memtable with next log file: 10813
>
>2018-07-11 16:43:22.994745 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327402994738, "job": 1124, "event": "flush_started", "num_memtables": 1, "num_entries": 538, "num_deletes": 251, "memory_usage": 638976}
>2018-07-11 16:43:22.994749 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1124] Level-0 flush table #10814: started
>2018-07-11 16:43:23.045254 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327403045240, "cf_name": "default", "job": 1124, "event": "table_file_creation", "file_number": 10814, "file_size": 603456, "table_properties": {"data_size": 597317, "index_size": 2680, "filter_size": 2540, "raw_key_size": 6965, "raw_average_key_size": 20, "raw_value_size": 590731, "raw_average_value_size": 1727, "num_data_blocks": 82, "num_entries": 342, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:43:23.045264 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1124] Level-0 flush table #10814: 603456 bytes OK
>2018-07-11 16:43:23.103628 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.061793) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10814 started
>2018-07-11 16:43:23.103636 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.103593) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10814: memtable #1 done
>2018-07-11 16:43:23.103639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.103608) EVENT_LOG_v1 {"time_micros": 1531327403103601, "job": 1124, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:43:23.103641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.103619) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:43:23.103659 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1124] Try to delete WAL files size 626519, prev total WAL file size 626519, number of live WAL files 2.
>
>2018-07-11 16:43:23.103950 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:43:23.104001 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.103994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:43:23.104010 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1125] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:43:23.104017 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1124 Base level 0, inputs: [10814(589KB)], [10812(16MB)]
>
>2018-07-11 16:43:23.104029 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327403104020, "job": 1125, "event": "compaction_started", "files_L0": [10814], "files_L1": [10812], "score": -1, "input_data_size": 17906398}
>2018-07-11 16:43:23.295999 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1125] Generated table #10815: 3233 keys, 15932396 bytes
>2018-07-11 16:43:23.296047 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327403296020, "cf_name": "default", "job": 1125, "event": "table_file_creation", "file_number": 10815, "file_size": 15932396, "table_properties": {"data_size": 15811802, "index_size": 68653, "filter_size": 51014, "raw_key_size": 66332, "raw_average_key_size": 20, "raw_value_size": 15717340, "raw_average_value_size": 4861, "num_data_blocks": 1989, "num_entries": 3233, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:43:23.329173 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1125] Compacted 1@0 + 1@1 files to L1 => 15932396 bytes
>2018-07-11 16:43:23.371053 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.370987) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 70.8 wr, level 1, files in(1, 1) out(1) MB in(0.6, 16.5) out(15.2), read-write-amplify(56.1) write-amplify(26.4) OK, records in: 3743, records dropped: 510
>
>2018-07-11 16:43:23.371062 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:43:23.371008) EVENT_LOG_v1 {"time_micros": 1531327403370999, "job": 1125, "event": "compaction_finished", "compaction_time_micros": 225128, "output_level": 1, "num_output_files": 1, "total_output_size": 15932396, "num_input_records": 3743, "num_output_records": 3233, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:43:23.371264 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327403371260, "job": 1125, "event": "table_file_deletion", "file_number": 10814}
>2018-07-11 16:43:23.374638 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327403374635, "job": 1125, "event": "table_file_deletion", "file_number": 10812}
>2018-07-11 16:43:50.856154 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 5,11)
>2018-07-11 16:43:51.006985 7f29149e4700  1 mon.magna102@0(leader).log v21736 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21736)
>2018-07-11 16:44:04.862113 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:44:05.017618 7f29149e4700  1 mon.magna102@0(leader).log v21737 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21737)
>2018-07-11 16:44:05.333410 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:44:10.865008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:44:11.035798 7f29149e4700  1 mon.magna102@0(leader).log v21738 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21738)
>2018-07-11 16:45:05.333536 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:45:26.898728 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:45:27.058344 7f29149e4700  1 mon.magna102@0(leader).log v21739 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21739)
>2018-07-11 16:45:34.902241 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:45:35.059047 7f29149e4700  1 mon.magna102@0(leader).log v21740 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21740)
>2018-07-11 16:45:44.906561 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:45:45.059914 7f29149e4700  1 mon.magna102@0(leader).log v21741 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21741)
>2018-07-11 16:45:52.943606 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:45:53.060590 7f29149e4700  1 mon.magna102@0(leader).log v21742 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21742)
>2018-07-11 16:46:05.333713 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:47:05.333880 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:47:06.943274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:47:07.100487 7f29149e4700  1 mon.magna102@0(leader).log v21743 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21743)
>2018-07-11 16:47:10.945203 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:47:11.092573 7f29149e4700  1 mon.magna102@0(leader).log v21744 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21744)
>2018-07-11 16:47:30.953695 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:47:31.103632 7f29149e4700  1 mon.magna102@0(leader).log v21745 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21745)
>2018-07-11 16:48:05.334057 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:48:46.986882 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:48:47.135342 7f29149e4700  1 mon.magna102@0(leader).log v21746 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21746)
>2018-07-11 16:48:54.990391 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:48:55.144513 7f29149e4700  1 mon.magna102@0(leader).log v21747 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21747)
>2018-07-11 16:49:05.334230 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:49:06.995474 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:49:07.145510 7f29149e4700  1 mon.magna102@0(leader).log v21748 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21748)
>2018-07-11 16:49:13.036450 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:49:13.154398 7f29149e4700  1 mon.magna102@0(leader).log v21749 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21749)
>2018-07-11 16:50:05.334415 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:50:28.136230 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10816. Immutable memtables: 0.
>
>2018-07-11 16:50:28.136273 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1126] Syncing log #10813
>2018-07-11 16:50:28.152677 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.136264) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:50:28.152688 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1126] Flushing memtable with next log file: 10816
>
>2018-07-11 16:50:28.152715 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828152707, "job": 1126, "event": "flush_started", "num_memtables": 1, "num_entries": 1851, "num_deletes": 250, "memory_usage": 3539592}
>2018-07-11 16:50:28.152718 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1126] Level-0 flush table #10817: started
>2018-07-11 16:50:28.211574 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828211560, "cf_name": "default", "job": 1126, "event": "table_file_creation", "file_number": 10817, "file_size": 1948043, "table_properties": {"data_size": 1932509, "index_size": 7944, "filter_size": 6670, "raw_key_size": 10955, "raw_average_key_size": 21, "raw_value_size": 1919637, "raw_average_value_size": 3698, "num_data_blocks": 248, "num_entries": 519, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:50:28.211597 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1126] Level-0 flush table #10817: 1948043 bytes OK
>2018-07-11 16:50:28.261578 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.228073) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10817 started
>2018-07-11 16:50:28.261586 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.261541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10817: memtable #1 done
>2018-07-11 16:50:28.261589 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.261556) EVENT_LOG_v1 {"time_micros": 1531327828261550, "job": 1126, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:50:28.261604 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.261568) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:50:28.261610 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1126] Try to delete WAL files size 3503785, prev total WAL file size 3503785, number of live WAL files 2.
>
>2018-07-11 16:50:28.262508 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:50:28.262571 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.262564) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:50:28.262581 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1127] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:50:28.262588 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1126 Base level 0, inputs: [10817(1902KB)], [10815(15MB)]
>
>2018-07-11 16:50:28.262601 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828262591, "job": 1127, "event": "compaction_started", "files_L0": [10817], "files_L1": [10815], "score": -1, "input_data_size": 17880439}
>2018-07-11 16:50:28.470685 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1127] Generated table #10818: 3450 keys, 17573432 bytes
>2018-07-11 16:50:28.470735 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828470707, "cf_name": "default", "job": 1127, "event": "table_file_creation", "file_number": 10818, "file_size": 17573432, "table_properties": {"data_size": 17441118, "index_size": 75326, "filter_size": 56061, "raw_key_size": 70610, "raw_average_key_size": 20, "raw_value_size": 17338961, "raw_average_value_size": 5025, "num_data_blocks": 2193, "num_entries": 3450, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:50:28.512204 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1127] Compacted 1@0 + 1@1 files to L1 => 17573432 bytes
>2018-07-11 16:50:28.545674 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.545605) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.07, MB/sec: 71.6 rd, 70.4 wr, level 1, files in(1, 1) out(1) MB in(1.9, 15.2) out(16.8), read-write-amplify(18.2) write-amplify(9.0) OK, records in: 3752, records dropped: 302
>
>2018-07-11 16:50:28.545683 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:50:28.545626) EVENT_LOG_v1 {"time_micros": 1531327828545616, "job": 1127, "event": "compaction_finished", "compaction_time_micros": 249586, "output_level": 1, "num_output_files": 1, "total_output_size": 17573432, "num_input_records": 3752, "num_output_records": 3450, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:50:28.546142 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828546138, "job": 1127, "event": "table_file_deletion", "file_number": 10817}
>2018-07-11 16:50:28.549108 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327828549105, "job": 1127, "event": "table_file_deletion", "file_number": 10815}
>2018-07-11 16:50:28.549224 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:50:31.032349 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:50:31.179949 7f29149e4700  1 mon.magna102@0(leader).log v21750 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21750)
>2018-07-11 16:50:35.033849 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:50:35.180703 7f29149e4700  1 mon.magna102@0(leader).log v21751 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21751)
>2018-07-11 16:50:51.040624 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:50:51.191626 7f29149e4700  1 mon.magna102@0(leader).log v21752 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21752)
>2018-07-11 16:50:58.103593 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,11 (REQUEST_SLOW)
>2018-07-11 16:50:58.175535 7f29149e4700  1 mon.magna102@0(leader).log v21753 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21753)
>2018-07-11 16:51:05.184729 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10819. Immutable memtables: 0.
>
>2018-07-11 16:51:05.184770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1128] Syncing log #10816
>2018-07-11 16:51:05.201324 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.184761) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:51:05.201336 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1128] Flushing memtable with next log file: 10819
>
>2018-07-11 16:51:05.201350 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865201343, "job": 1128, "event": "flush_started", "num_memtables": 1, "num_entries": 421, "num_deletes": 251, "memory_usage": 384656}
>2018-07-11 16:51:05.201353 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1128] Level-0 flush table #10820: started
>2018-07-11 16:51:05.243595 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865243582, "cf_name": "default", "job": 1128, "event": "table_file_creation", "file_number": 10820, "file_size": 364601, "table_properties": {"data_size": 360348, "index_size": 1559, "filter_size": 1775, "raw_key_size": 6289, "raw_average_key_size": 20, "raw_value_size": 355021, "raw_average_value_size": 1145, "num_data_blocks": 48, "num_entries": 310, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:51:05.243606 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1128] Level-0 flush table #10820: 364601 bytes OK
>2018-07-11 16:51:05.293628 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.260144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10820 started
>2018-07-11 16:51:05.293637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.293594) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10820: memtable #1 done
>2018-07-11 16:51:05.293639 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.293608) EVENT_LOG_v1 {"time_micros": 1531327865293602, "job": 1128, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:51:05.293641 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.293620) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:51:05.293659 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1128] Try to delete WAL files size 374404, prev total WAL file size 374404, number of live WAL files 2.
>
>2018-07-11 16:51:05.293817 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:51:05.293867 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.293860) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:51:05.293877 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1129] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:51:05.293883 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1128 Base level 0, inputs: [10820(356KB)], [10818(16MB)]
>
>2018-07-11 16:51:05.293895 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865293886, "job": 1129, "event": "compaction_started", "files_L0": [10820], "files_L1": [10818], "score": -1, "input_data_size": 17938033}
>2018-07-11 16:51:05.334598 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8832 MB, avail 861 GB
>2018-07-11 16:51:05.485990 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1129] Generated table #10821: 3250 keys, 15955197 bytes
>2018-07-11 16:51:05.486032 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865486005, "cf_name": "default", "job": 1129, "event": "table_file_creation", "file_number": 10821, "file_size": 15955197, "table_properties": {"data_size": 15834510, "index_size": 68665, "filter_size": 51095, "raw_key_size": 66652, "raw_average_key_size": 20, "raw_value_size": 15739798, "raw_average_value_size": 4843, "num_data_blocks": 1989, "num_entries": 3250, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:51:05.527571 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1129] Compacted 1@0 + 1@1 files to L1 => 15955197 bytes
>2018-07-11 16:51:05.561066 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.561000) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.8 rd, 68.3 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.8) out(15.2), read-write-amplify(93.0) write-amplify(43.8) OK, records in: 3760, records dropped: 510
>
>2018-07-11 16:51:05.561075 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:51:05.561022) EVENT_LOG_v1 {"time_micros": 1531327865561012, "job": 1129, "event": "compaction_finished", "compaction_time_micros": 233660, "output_level": 1, "num_output_files": 1, "total_output_size": 15955197, "num_input_records": 3760, "num_output_records": 3250, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:51:05.561237 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865561232, "job": 1129, "event": "table_file_deletion", "file_number": 10820}
>2018-07-11 16:51:05.564612 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531327865564609, "job": 1129, "event": "table_file_deletion", "file_number": 10818}
>2018-07-11 16:51:47.064703 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 16:51:47.207613 7f29149e4700  1 mon.magna102@0(leader).log v21754 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21754)
>2018-07-11 16:51:49.065496 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 16:51:49.216258 7f29149e4700  1 mon.magna102@0(leader).log v21755 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21755)
>2018-07-11 16:51:54.249978 7f29149e4700  1 mon.magna102@0(leader).log v21756 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21756)
>2018-07-11 16:51:56.400294 7f29149e4700  1 mon.magna102@0(leader).log v21757 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21757)
>2018-07-11 16:52:05.072422 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 16:52:05.226236 7f29149e4700  1 mon.magna102@0(leader).log v21758 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21758)
>2018-07-11 16:52:05.334798 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:52:13.109979 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 16:52:13.226998 7f29149e4700  1 mon.magna102@0(leader).log v21759 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21759)
>2018-07-11 16:53:05.334973 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:53:21.105365 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,11)
>2018-07-11 16:53:21.258072 7f29149e4700  1 mon.magna102@0(leader).log v21760 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21760)
>2018-07-11 16:53:41.113909 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 16:53:41.267974 7f29149e4700  1 mon.magna102@0(leader).log v21761 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21761)
>2018-07-11 16:54:05.335111 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:54:57.316209 7f29149e4700  1 mon.magna102@0(leader).log v21762 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21762)
>2018-07-11 16:55:01.148911 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 0,11)
>2018-07-11 16:55:01.309479 7f29149e4700  1 mon.magna102@0(leader).log v21763 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21763)
>2018-07-11 16:55:05.335291 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:55:21.157576 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 16:55:21.311238 7f29149e4700  1 mon.magna102@0(leader).log v21764 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21764)
>2018-07-11 16:56:05.335465 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:56:35.189712 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:56:35.359375 7f29149e4700  1 mon.magna102@0(leader).log v21765 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21765)
>2018-07-11 16:56:37.190471 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:56:37.368004 7f29149e4700  1 mon.magna102@0(leader).log v21766 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21766)
>2018-07-11 16:56:57.199109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:56:57.352907 7f29149e4700  1 mon.magna102@0(leader).log v21767 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21767)
>2018-07-11 16:57:03.236490 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 16:57:03.353522 7f29149e4700  1 mon.magna102@0(leader).log v21768 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21768)
>2018-07-11 16:57:05.335651 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:58:05.335828 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 16:58:17.234176 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 16:58:17.384998 7f29149e4700  1 mon.magna102@0(leader).log v21769 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21769)
>2018-07-11 16:58:21.236045 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 0)
>2018-07-11 16:58:21.410535 7f29149e4700  1 mon.magna102@0(leader).log v21770 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21770)
>2018-07-11 16:58:23.378050 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10822. Immutable memtables: 0.
>
>2018-07-11 16:58:23.378094 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1130] Syncing log #10819
>2018-07-11 16:58:23.395499 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.378084) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:58:23.395510 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1130] Flushing memtable with next log file: 10822
>
>2018-07-11 16:58:23.395537 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303395529, "job": 1130, "event": "flush_started", "num_memtables": 1, "num_entries": 2174, "num_deletes": 500, "memory_usage": 3717896}
>2018-07-11 16:58:23.395540 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1130] Level-0 flush table #10823: started
>2018-07-11 16:58:23.462715 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303462700, "cf_name": "default", "job": 1130, "event": "table_file_creation", "file_number": 10823, "file_size": 3526472, "table_properties": {"data_size": 3497318, "index_size": 15832, "filter_size": 12397, "raw_key_size": 20711, "raw_average_key_size": 20, "raw_value_size": 3472328, "raw_average_value_size": 3479, "num_data_blocks": 476, "num_entries": 998, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 16:58:23.462726 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1130] Level-0 flush table #10823: 3526472 bytes OK
>2018-07-11 16:58:23.512747 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.479236) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10823 started
>2018-07-11 16:58:23.512755 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.512709) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10823: memtable #1 done
>2018-07-11 16:58:23.512757 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.512724) EVENT_LOG_v1 {"time_micros": 1531328303512717, "job": 1130, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:58:23.512759 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.512737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:58:23.512777 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1130] Try to delete WAL files size 3673261, prev total WAL file size 3673261, number of live WAL files 2.
>
>2018-07-11 16:58:23.513753 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:58:23.513948 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 16:58:23.513954 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 90330.4 total, 975.5 interval
>Cumulative writes: 116K writes, 546K keys, 116K commit groups, 1.0 writes per commit group, ingest: 0.90 GB, 0.01 MB/s
>Cumulative WAL: 116K writes, 116K syncs, 1.00 writes per sync, written: 0.90 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1058 writes, 4984 keys, 1058 commit groups, 1.0 writes per commit group, ingest: 7.80 MB, 0.01 MB/s
>Interval WAL: 1058 writes, 1058 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.36 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       566    0.079       0      0
>  L1      1/1   15.22 MB   0.0      8.9     0.8      8.2       8.2     -0.0       0.0  10.8     72.6     66.5       126       563    0.224   2024K   243K
> Sum      2/2   18.58 MB   0.0      8.9     0.8      8.2       8.9      0.8       0.0  11.8     53.6     53.6       171      1129    0.151   2024K   243K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.4     55.6     56.9         1         8    0.153     14K   1640
>Uptime(secs): 90330.4 total, 90330.4 interval
>Flush(GB): cumulative 0.759, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.93 GB write, 0.10 MB/s write, 8.93 GB read, 0.10 MB/s read, 170.6 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.36 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       566    0.079       0      0
>  L1      1/1   15.22 MB   0.0      8.9     0.8      8.2       8.2     -0.0       0.0  10.8     72.6     66.5       126       563    0.224   2024K   243K
> Sum      2/2   18.58 MB   0.0      8.9     0.8      8.2       8.9      0.8       0.0  11.8     53.6     53.6       171      1129    0.151   2024K   243K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 90330.4 total, 90330.4 interval
>Flush(GB): cumulative 0.759, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 8.93 GB write, 0.10 MB/s write, 8.93 GB read, 0.10 MB/s read, 170.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 16:58:23.513975 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.513969) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 16:58:23.513980 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1131] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:58:23.513986 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1130 Base level 0, inputs: [10823(3443KB)], [10821(15MB)]
>
>2018-07-11 16:58:23.513998 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303513989, "job": 1131, "event": "compaction_started", "files_L0": [10823], "files_L1": [10821], "score": -1, "input_data_size": 19481669}
>2018-07-11 16:58:23.713504 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1131] Generated table #10824: 3240 keys, 17003412 bytes
>2018-07-11 16:58:23.713543 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303713527, "cf_name": "default", "job": 1131, "event": "table_file_creation", "file_number": 10824, "file_size": 17003412, "table_properties": {"data_size": 16872661, "index_size": 75294, "filter_size": 54530, "raw_key_size": 66657, "raw_average_key_size": 20, "raw_value_size": 16773445, "raw_average_value_size": 5176, "num_data_blocks": 2198, "num_entries": 3240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:58:23.746500 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1131] Compacted 1@0 + 1@1 files to L1 => 17003412 bytes
>2018-07-11 16:58:23.780159 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.780091) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 83.8 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(3.4, 15.2) out(16.2), read-write-amplify(10.3) write-amplify(4.8) OK, records in: 4248, records dropped: 1008
>
>2018-07-11 16:58:23.780168 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:23.780112) EVENT_LOG_v1 {"time_micros": 1531328303780103, "job": 1131, "event": "compaction_finished", "compaction_time_micros": 232485, "output_level": 1, "num_output_files": 1, "total_output_size": 17003412, "num_input_records": 4248, "num_output_records": 3240, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:58:23.780944 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303780940, "job": 1131, "event": "table_file_deletion", "file_number": 10823}
>2018-07-11 16:58:23.783963 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328303783960, "job": 1131, "event": "table_file_deletion", "file_number": 10821}
>2018-07-11 16:58:23.784029 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:58:41.244699 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 16:58:41.415419 7f29149e4700  1 mon.magna102@0(leader).log v21771 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21771)
>2018-07-11 16:58:47.407803 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10825. Immutable memtables: 0.
>
>2018-07-11 16:58:47.407843 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1132] Syncing log #10822
>2018-07-11 16:58:47.424437 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.407835) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:58:47.424448 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1132] Flushing memtable with next log file: 10825
>
>2018-07-11 16:58:47.424474 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327424467, "job": 1132, "event": "flush_started", "num_memtables": 1, "num_entries": 348, "num_deletes": 251, "memory_usage": 221144}
>2018-07-11 16:58:47.424478 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1132] Level-0 flush table #10826: started
>2018-07-11 16:58:47.541614 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327541602, "cf_name": "default", "job": 1132, "event": "table_file_creation", "file_number": 10826, "file_size": 207140, "table_properties": {"data_size": 204035, "index_size": 902, "filter_size": 1284, "raw_key_size": 5812, "raw_average_key_size": 20, "raw_value_size": 199515, "raw_average_value_size": 695, "num_data_blocks": 28, "num_entries": 287, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 16:58:47.541625 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1132] Level-0 flush table #10826: 207140 bytes OK
>2018-07-11 16:58:47.591655 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.558153) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10826 started
>2018-07-11 16:58:47.591664 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.591620) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10826: memtable #1 done
>2018-07-11 16:58:47.591679 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.591635) EVENT_LOG_v1 {"time_micros": 1531328327591629, "job": 1132, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:58:47.591681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.591646) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:58:47.591686 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1132] Try to delete WAL files size 212013, prev total WAL file size 212013, number of live WAL files 2.
>
>2018-07-11 16:58:47.591813 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:58:47.591862 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.591855) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 16:58:47.591871 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1133] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:58:47.591878 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1132 Base level 0, inputs: [10826(202KB)], [10824(16MB)]
>
>2018-07-11 16:58:47.591890 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327591881, "job": 1133, "event": "compaction_started", "files_L0": [10826], "files_L1": [10824], "score": -1, "input_data_size": 17210552}
>2018-07-11 16:58:47.775692 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1133] Generated table #10827: 3017 keys, 15248023 bytes
>2018-07-11 16:58:47.775735 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327775709, "cf_name": "default", "job": 1133, "event": "table_file_creation", "file_number": 10827, "file_size": 15248023, "table_properties": {"data_size": 15130032, "index_size": 67952, "filter_size": 49112, "raw_key_size": 62222, "raw_average_key_size": 20, "raw_value_size": 15039069, "raw_average_value_size": 4984, "num_data_blocks": 1974, "num_entries": 3017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:58:47.817227 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1133] Compacted 1@0 + 1@1 files to L1 => 15248023 bytes
>2018-07-11 16:58:47.850732 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.850662) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.2) out(14.5), read-write-amplify(156.7) write-amplify(73.6) OK, records in: 3527, records dropped: 510
>
>2018-07-11 16:58:47.850741 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:47.850683) EVENT_LOG_v1 {"time_micros": 1531328327850674, "job": 1133, "event": "compaction_finished", "compaction_time_micros": 225321, "output_level": 1, "num_output_files": 1, "total_output_size": 15248023, "num_input_records": 3527, "num_output_records": 3017, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:58:47.850864 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327850860, "job": 1133, "event": "table_file_deletion", "file_number": 10826}
>2018-07-11 16:58:47.854033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328327854030, "job": 1133, "event": "table_file_deletion", "file_number": 10824}
>2018-07-11 16:58:48.368219 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10828. Immutable memtables: 0.
>
>2018-07-11 16:58:48.368287 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1134] Syncing log #10825
>2018-07-11 16:58:48.384856 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.368278) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 16:58:48.384869 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1134] Flushing memtable with next log file: 10828
>
>2018-07-11 16:58:48.384882 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328384875, "job": 1134, "event": "flush_started", "num_memtables": 1, "num_entries": 255, "num_deletes": 250, "memory_usage": 20248}
>2018-07-11 16:58:48.384885 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1134] Level-0 flush table #10829: started
>2018-07-11 16:58:48.426937 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328426924, "cf_name": "default", "job": 1134, "event": "table_file_creation", "file_number": 10829, "file_size": 14277, "table_properties": {"data_size": 12637, "index_size": 53, "filter_size": 673, "raw_key_size": 5626, "raw_average_key_size": 22, "raw_value_size": 9205, "raw_average_value_size": 36, "num_data_blocks": 2, "num_entries": 255, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 16:58:48.426946 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1134] Level-0 flush table #10829: 14277 bytes OK
>2018-07-11 16:58:48.477023 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.443501) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10829 started
>2018-07-11 16:58:48.477031 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.476989) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10829: memtable #1 done
>2018-07-11 16:58:48.477033 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.477003) EVENT_LOG_v1 {"time_micros": 1531328328476997, "job": 1134, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 16:58:48.477035 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.477014) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 16:58:48.477052 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1134] Try to delete WAL files size 13331, prev total WAL file size 13331, number of live WAL files 2.
>
>2018-07-11 16:58:48.477148 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:58:48.477203 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.477197) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 16:58:48.477212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1135] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 16:58:48.477218 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1134 Base level 0, inputs: [10829(13KB)], [10827(14MB)]
>
>2018-07-11 16:58:48.477231 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328477221, "job": 1135, "event": "compaction_started", "files_L0": [10829], "files_L1": [10827], "score": -1, "input_data_size": 15262300}
>2018-07-11 16:58:48.636010 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1135] Generated table #10830: 2768 keys, 13544386 bytes
>2018-07-11 16:58:48.636059 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328636032, "cf_name": "default", "job": 1135, "event": "table_file_creation", "file_number": 10830, "file_size": 13544386, "table_properties": {"data_size": 13440624, "index_size": 59264, "filter_size": 43571, "raw_key_size": 56742, "raw_average_key_size": 20, "raw_value_size": 13359374, "raw_average_value_size": 4826, "num_data_blocks": 1725, "num_entries": 2768, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 16:58:48.669220 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1135] Compacted 1@0 + 1@1 files to L1 => 13544386 bytes
>2018-07-11 16:58:48.702720 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.702653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.5 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.0, 14.5) out(12.9), read-write-amplify(2017.7) write-amplify(948.7) OK, records in: 3272, records dropped: 504
>
>2018-07-11 16:58:48.702728 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-16:58:48.702675) EVENT_LOG_v1 {"time_micros": 1531328328702665, "job": 1135, "event": "compaction_finished", "compaction_time_micros": 191973, "output_level": 1, "num_output_files": 1, "total_output_size": 13544386, "num_input_records": 3272, "num_output_records": 2768, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 16:58:48.702824 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328702820, "job": 1135, "event": "table_file_deletion", "file_number": 10829}
>2018-07-11 16:58:48.705781 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328328705778, "job": 1135, "event": "table_file_deletion", "file_number": 10827}
>2018-07-11 16:58:48.705894 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 16:59:05.336019 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 16:59:55.276747 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 16:59:55.451087 7f29149e4700  1 mon.magna102@0(leader).log v21772 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21772)
>2018-07-11 16:59:57.277513 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 16:59:57.460935 7f29149e4700  1 mon.magna102@0(leader).log v21773 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21773)
>2018-07-11 17:00:05.336215 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:00:15.285546 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 17:00:15.437785 7f29149e4700  1 mon.magna102@0(leader).log v21774 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21774)
>2018-07-11 17:00:23.329773 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 17:00:23.446896 7f29149e4700  1 mon.magna102@0(leader).log v21775 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21775)
>2018-07-11 17:01:05.343069 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:01:35.321032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:01:35.495434 7f29149e4700  1 mon.magna102@0(leader).log v21776 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21776)
>2018-07-11 17:01:37.321738 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 17:01:37.512413 7f29149e4700  1 mon.magna102@0(leader).log v21777 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21777)
>2018-07-11 17:01:51.327767 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:01:51.488653 7f29149e4700  1 mon.magna102@0(leader).log v21778 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21778)
>2018-07-11 17:01:57.272320 7f29149e4700  1 mon.magna102@0(leader).log v21779 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21779)
>2018-07-11 17:01:58.363822 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 17:01:59.406042 7f29149e4700  1 mon.magna102@0(leader).log v21780 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21780)
>2018-07-11 17:02:05.366070 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:03:05.395117 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:03:09.361741 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:03:09.529109 7f29149e4700  1 mon.magna102@0(leader).log v21781 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21781)
>2018-07-11 17:03:11.362606 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 17:03:11.537767 7f29149e4700  1 mon.magna102@0(leader).log v21782 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21782)
>2018-07-11 17:03:31.371175 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:03:31.523962 7f29149e4700  1 mon.magna102@0(leader).log v21783 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21783)
>2018-07-11 17:03:38.415225 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 17:03:38.499544 7f29149e4700  1 mon.magna102@0(leader).log v21784 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21784)
>2018-07-11 17:04:05.417963 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:04:49.404530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:04:49.564210 7f29149e4700  1 mon.magna102@0(leader).log v21785 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21785)
>2018-07-11 17:04:51.405345 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 17:04:51.572884 7f29149e4700  1 mon.magna102@0(leader).log v21786 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21786)
>2018-07-11 17:05:05.448647 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:05:09.412970 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 17:05:09.590947 7f29149e4700  1 mon.magna102@0(leader).log v21787 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21787)
>2018-07-11 17:05:18.457901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 17:05:18.574980 7f29149e4700  1 mon.magna102@0(leader).log v21788 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21788)
>2018-07-11 17:06:05.471299 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:06:29.447584 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:06:29.606622 7f29149e4700  1 mon.magna102@0(leader).log v21789 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21789)
>2018-07-11 17:06:29.682019 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10831. Immutable memtables: 0.
>
>2018-07-11 17:06:29.682063 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1136] Syncing log #10828
>2018-07-11 17:06:29.698585 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.682053) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:06:29.698596 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1136] Flushing memtable with next log file: 10831
>
>2018-07-11 17:06:29.698623 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328789698614, "job": 1136, "event": "flush_started", "num_memtables": 1, "num_entries": 2017, "num_deletes": 251, "memory_usage": 3913272}
>2018-07-11 17:06:29.698626 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1136] Level-0 flush table #10832: started
>2018-07-11 17:06:29.774157 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328789774142, "cf_name": "default", "job": 1136, "event": "table_file_creation", "file_number": 10832, "file_size": 3718357, "table_properties": {"data_size": 3688377, "index_size": 16675, "filter_size": 12380, "raw_key_size": 16047, "raw_average_key_size": 20, "raw_value_size": 3665643, "raw_average_value_size": 4723, "num_data_blocks": 501, "num_entries": 776, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:06:29.774169 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1136] Level-0 flush table #10832: 3718357 bytes OK
>2018-07-11 17:06:29.824138 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.790653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10832 started
>2018-07-11 17:06:29.824147 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.824098) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10832: memtable #1 done
>2018-07-11 17:06:29.824161 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.824113) EVENT_LOG_v1 {"time_micros": 1531328789824106, "job": 1136, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:06:29.824164 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.824127) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:06:29.824170 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1136] Try to delete WAL files size 3873904, prev total WAL file size 3873904, number of live WAL files 2.
>
>2018-07-11 17:06:29.825125 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:06:29.825187 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:06:29.825172) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:06:29.825197 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1137] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:06:29.825203 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1136 Base level 0, inputs: [10832(3631KB)], [10830(12MB)]
>
>2018-07-11 17:06:29.825216 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328789825206, "job": 1137, "event": "compaction_started", "files_L0": [10832], "files_L1": [10830], "score": -1, "input_data_size": 17262743}
>2018-07-11 17:06:30.017770 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1137] Generated table #10833: 3034 keys, 15294696 bytes
>2018-07-11 17:06:30.017815 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328790017788, "cf_name": "default", "job": 1137, "event": "table_file_creation", "file_number": 10833, "file_size": 15294696, "table_properties": {"data_size": 15176561, "index_size": 67967, "filter_size": 49241, "raw_key_size": 62542, "raw_average_key_size": 20, "raw_value_size": 15085349, "raw_average_value_size": 4972, "num_data_blocks": 1974, "num_entries": 3034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:06:30.059284 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1137] Compacted 1@0 + 1@1 files to L1 => 15294696 bytes
>2018-07-11 17:06:30.101155 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:06:30.101080) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.8 rd, 65.3 wr, level 1, files in(1, 1) out(1) MB in(3.5, 12.9) out(14.6), read-write-amplify(8.8) write-amplify(4.1) OK, records in: 3544, records dropped: 510
>
>2018-07-11 17:06:30.101164 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:06:30.101102) EVENT_LOG_v1 {"time_micros": 1531328790101092, "job": 1137, "event": "compaction_finished", "compaction_time_micros": 234051, "output_level": 1, "num_output_files": 1, "total_output_size": 15294696, "num_input_records": 3544, "num_output_records": 3034, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:06:30.101974 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328790101969, "job": 1137, "event": "table_file_deletion", "file_number": 10832}
>2018-07-11 17:06:30.104537 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328790104534, "job": 1137, "event": "table_file_deletion", "file_number": 10830}
>2018-07-11 17:06:31.448684 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 17:06:31.601897 7f29149e4700  1 mon.magna102@0(leader).log v21790 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21790)
>2018-07-11 17:06:49.456452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 0 (REQUEST_SLOW)
>2018-07-11 17:06:49.603410 7f29149e4700  1 mon.magna102@0(leader).log v21791 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21791)
>2018-07-11 17:06:58.495317 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 0,11 (REQUEST_SLOW)
>2018-07-11 17:06:58.595798 7f29149e4700  1 mon.magna102@0(leader).log v21792 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21792)
>2018-07-11 17:07:05.496241 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:07:08.571881 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10834. Immutable memtables: 0.
>
>2018-07-11 17:07:08.571923 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1138] Syncing log #10831
>2018-07-11 17:07:08.588463 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.571914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:07:08.588474 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1138] Flushing memtable with next log file: 10834
>
>2018-07-11 17:07:08.588500 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828588492, "job": 1138, "event": "flush_started", "num_memtables": 1, "num_entries": 412, "num_deletes": 250, "memory_usage": 367248}
>2018-07-11 17:07:08.588503 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1138] Level-0 flush table #10835: started
>2018-07-11 17:07:08.622307 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828622294, "cf_name": "default", "job": 1138, "event": "table_file_creation", "file_number": 10835, "file_size": 258309, "table_properties": {"data_size": 254905, "index_size": 1053, "filter_size": 1432, "raw_key_size": 6424, "raw_average_key_size": 21, "raw_value_size": 250165, "raw_average_value_size": 853, "num_data_blocks": 33, "num_entries": 293, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:07:08.622317 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1138] Level-0 flush table #10835: 258309 bytes OK
>2018-07-11 17:07:08.672347 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.638843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10835 started
>2018-07-11 17:07:08.672355 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.672312) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10835: memtable #1 done
>2018-07-11 17:07:08.672357 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.672327) EVENT_LOG_v1 {"time_micros": 1531328828672320, "job": 1138, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:07:08.672373 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.672338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:07:08.672379 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1138] Try to delete WAL files size 357476, prev total WAL file size 357476, number of live WAL files 2.
>
>2018-07-11 17:07:08.672535 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:07:08.672585 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.672578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:07:08.672594 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1139] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:07:08.672600 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1138 Base level 0, inputs: [10835(252KB)], [10833(14MB)]
>
>2018-07-11 17:07:08.672613 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828672604, "job": 1139, "event": "compaction_started", "files_L0": [10835], "files_L1": [10833], "score": -1, "input_data_size": 15553005}
>2018-07-11 17:07:08.848095 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1139] Generated table #10836: 2832 keys, 13923082 bytes
>2018-07-11 17:07:08.848144 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828848116, "cf_name": "default", "job": 1139, "event": "table_file_creation", "file_number": 10836, "file_size": 13923082, "table_properties": {"data_size": 13816636, "index_size": 60771, "filter_size": 44748, "raw_key_size": 58043, "raw_average_key_size": 20, "raw_value_size": 13733422, "raw_average_value_size": 4849, "num_data_blocks": 1769, "num_entries": 2832, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:07:08.881274 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1139] Compacted 1@0 + 1@1 files to L1 => 13923082 bytes
>2018-07-11 17:07:08.914791 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.914725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.5 rd, 66.7 wr, level 1, files in(1, 1) out(1) MB in(0.2, 14.6) out(13.3), read-write-amplify(114.1) write-amplify(53.9) OK, records in: 3327, records dropped: 495
>
>2018-07-11 17:07:08.914803 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:07:08.914747) EVENT_LOG_v1 {"time_micros": 1531328828914737, "job": 1139, "event": "compaction_finished", "compaction_time_micros": 208644, "output_level": 1, "num_output_files": 1, "total_output_size": 13923082, "num_input_records": 3327, "num_output_records": 2832, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:07:08.914935 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828914931, "job": 1139, "event": "table_file_deletion", "file_number": 10835}
>2018-07-11 17:07:08.917825 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531328828917822, "job": 1139, "event": "table_file_deletion", "file_number": 10833}
>2018-07-11 17:07:08.917938 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:07:29.474283 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 11 (REQUEST_SLOW)
>2018-07-11 17:07:29.643474 7f29149e4700  1 mon.magna102@0(leader).log v21793 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21793)
>2018-07-11 17:07:31.475231 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 11)
>2018-07-11 17:07:31.627095 7f29149e4700  1 mon.magna102@0(leader).log v21794 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21794)
>2018-07-11 17:07:33.327219 7f29149e4700  1 mon.magna102@0(leader).log v21795 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21795)
>2018-07-11 17:07:44.619936 7f29149e4700  1 mon.magna102@0(leader).log v21796 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21796)
>2018-07-11 17:07:47.203550 7f29149e4700  1 mon.magna102@0(leader).log v21797 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21797)
>2018-07-11 17:08:05.529937 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:09:05.552172 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:10:05.582743 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:11:05.605058 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:12:05.635447 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:13:05.656148 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:14:05.678214 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8793 MB, avail 861 GB
>2018-07-11 17:14:31.790092 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10837. Immutable memtables: 0.
>
>2018-07-11 17:14:31.790136 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1140] Syncing log #10834
>2018-07-11 17:14:31.806650 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.790126) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:14:31.806661 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1140] Flushing memtable with next log file: 10837
>
>2018-07-11 17:14:31.806687 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329271806679, "job": 1140, "event": "flush_started", "num_memtables": 1, "num_entries": 1850, "num_deletes": 251, "memory_usage": 3428632}
>2018-07-11 17:14:31.806691 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1140] Level-0 flush table #10838: started
>2018-07-11 17:14:31.873741 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329271873726, "cf_name": "default", "job": 1140, "event": "table_file_creation", "file_number": 10838, "file_size": 3272760, "table_properties": {"data_size": 3245612, "index_size": 15138, "filter_size": 11086, "raw_key_size": 14876, "raw_average_key_size": 20, "raw_value_size": 3224771, "raw_average_value_size": 4485, "num_data_blocks": 456, "num_entries": 719, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:14:31.873764 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1140] Level-0 flush table #10838: 3272760 bytes OK
>2018-07-11 17:14:31.923874 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.890393) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10838 started
>2018-07-11 17:14:31.923882 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.923836) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10838: memtable #1 done
>2018-07-11 17:14:31.923885 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.923851) EVENT_LOG_v1 {"time_micros": 1531329271923844, "job": 1140, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:14:31.923887 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.923864) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:14:31.923906 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1140] Try to delete WAL files size 3391526, prev total WAL file size 3391526, number of live WAL files 2.
>
>2018-07-11 17:14:31.924811 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:14:31.925004 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 17:14:31.925011 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 91298.8 total, 968.4 interval
>Cumulative writes: 117K writes, 551K keys, 117K commit groups, 1.0 writes per commit group, ingest: 0.91 GB, 0.01 MB/s
>Cumulative WAL: 117K writes, 117K syncs, 1.00 writes per sync, written: 0.91 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1032 writes, 4882 keys, 1032 commit groups, 1.0 writes per commit group, ingest: 7.48 MB, 0.01 MB/s
>Interval WAL: 1032 writes, 1032 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.12 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       571    0.079       0      0
>  L1      1/1   13.28 MB   0.0      9.0     0.8      8.2       8.2     -0.0       0.0  10.8     72.6     66.5       127       568    0.223   2042K   246K
> Sum      2/2   16.40 MB   0.0      9.0     0.8      8.2       9.0      0.8       0.0  11.8     53.6     53.6       172      1139    0.151   2042K   246K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.0     53.5     52.1         2        10    0.151     17K   3027
>Uptime(secs): 91298.8 total, 91298.8 interval
>Flush(GB): cumulative 0.766, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.01 GB write, 0.10 MB/s write, 9.00 GB read, 0.10 MB/s read, 172.1 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.12 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       571    0.079       0      0
>  L1      1/1   13.28 MB   0.0      9.0     0.8      8.2       8.2     -0.0       0.0  10.8     72.6     66.5       127       568    0.223   2042K   246K
> Sum      2/2   16.40 MB   0.0      9.0     0.8      8.2       9.0      0.8       0.0  11.8     53.6     53.6       172      1139    0.151   2042K   246K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 91298.8 total, 91298.8 interval
>Flush(GB): cumulative 0.766, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.01 GB write, 0.10 MB/s write, 9.00 GB read, 0.10 MB/s read, 172.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 17:14:31.925032 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:14:31.925025) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:14:31.925037 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1141] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:14:31.925042 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1140 Base level 0, inputs: [10838(3196KB)], [10836(13MB)]
>
>2018-07-11 17:14:31.925055 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329271925045, "job": 1141, "event": "compaction_started", "files_L0": [10838], "files_L1": [10836], "score": -1, "input_data_size": 17195842}
>2018-07-11 17:14:32.116252 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1141] Generated table #10839: 3041 keys, 15221749 bytes
>2018-07-11 17:14:32.116289 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329272116274, "cf_name": "default", "job": 1141, "event": "table_file_creation", "file_number": 10839, "file_size": 15221749, "table_properties": {"data_size": 15103776, "index_size": 67937, "filter_size": 49109, "raw_key_size": 62672, "raw_average_key_size": 20, "raw_value_size": 15012487, "raw_average_value_size": 4936, "num_data_blocks": 1973, "num_entries": 3041, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:14:32.149415 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1141] Compacted 1@0 + 1@1 files to L1 => 15221749 bytes
>2018-07-11 17:14:32.182927 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:14:32.182857) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.6 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(3.1, 13.3) out(14.5), read-write-amplify(9.9) write-amplify(4.7) OK, records in: 3551, records dropped: 510
>
>2018-07-11 17:14:32.182937 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:14:32.182878) EVENT_LOG_v1 {"time_micros": 1531329272182869, "job": 1141, "event": "compaction_finished", "compaction_time_micros": 224343, "output_level": 1, "num_output_files": 1, "total_output_size": 15221749, "num_input_records": 3551, "num_output_records": 3041, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:14:32.183697 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329272183693, "job": 1141, "event": "table_file_deletion", "file_number": 10838}
>2018-07-11 17:14:32.186394 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329272186391, "job": 1141, "event": "table_file_deletion", "file_number": 10836}
>2018-07-11 17:15:05.702238 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:15:28.797146 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10840. Immutable memtables: 0.
>
>2018-07-11 17:15:28.797194 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1142] Syncing log #10837
>2018-07-11 17:15:28.813713 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.797185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:15:28.813724 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1142] Flushing memtable with next log file: 10840
>
>2018-07-11 17:15:28.813750 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329328813742, "job": 1142, "event": "flush_started", "num_memtables": 1, "num_entries": 451, "num_deletes": 250, "memory_usage": 437552}
>2018-07-11 17:15:28.813754 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1142] Level-0 flush table #10841: started
>2018-07-11 17:15:28.855919 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329328855905, "cf_name": "default", "job": 1142, "event": "table_file_creation", "file_number": 10841, "file_size": 264344, "table_properties": {"data_size": 260778, "index_size": 1177, "filter_size": 1470, "raw_key_size": 6370, "raw_average_key_size": 21, "raw_value_size": 256000, "raw_average_value_size": 879, "num_data_blocks": 37, "num_entries": 291, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:15:28.855929 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1142] Level-0 flush table #10841: 264344 bytes OK
>2018-07-11 17:15:28.905939 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.872447) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10841 started
>2018-07-11 17:15:28.905947 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.905904) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10841: memtable #1 done
>2018-07-11 17:15:28.905949 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.905918) EVENT_LOG_v1 {"time_micros": 1531329328905912, "job": 1142, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:15:28.905951 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.905929) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:15:28.905970 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1142] Try to delete WAL files size 426863, prev total WAL file size 426863, number of live WAL files 2.
>
>2018-07-11 17:15:28.906144 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:15:28.906203 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:15:28.906196) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:15:28.906212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1143] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:15:28.906219 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1142 Base level 0, inputs: [10841(258KB)], [10839(14MB)]
>
>2018-07-11 17:15:28.906232 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329328906222, "job": 1143, "event": "compaction_started", "files_L0": [10841], "files_L1": [10839], "score": -1, "input_data_size": 15486093}
>2018-07-11 17:15:29.089842 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1143] Generated table #10842: 2848 keys, 13933697 bytes
>2018-07-11 17:15:29.089887 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329329089872, "cf_name": "default", "job": 1143, "event": "table_file_creation", "file_number": 10842, "file_size": 13933697, "table_properties": {"data_size": 13826744, "index_size": 61146, "filter_size": 44880, "raw_key_size": 58368, "raw_average_key_size": 20, "raw_value_size": 13743040, "raw_average_value_size": 4825, "num_data_blocks": 1780, "num_entries": 2848, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:15:29.123087 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1143] Compacted 1@0 + 1@1 files to L1 => 13933697 bytes
>2018-07-11 17:15:29.156620 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:15:29.156548) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 71.4 rd, 64.3 wr, level 1, files in(1, 1) out(1) MB in(0.3, 14.5) out(13.3), read-write-amplify(111.3) write-amplify(52.7) OK, records in: 3332, records dropped: 484
>
>2018-07-11 17:15:29.156630 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:15:29.156572) EVENT_LOG_v1 {"time_micros": 1531329329156562, "job": 1143, "event": "compaction_finished", "compaction_time_micros": 216839, "output_level": 1, "num_output_files": 1, "total_output_size": 13933697, "num_input_records": 3332, "num_output_records": 2848, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:15:29.156756 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329329156752, "job": 1143, "event": "table_file_deletion", "file_number": 10841}
>2018-07-11 17:15:29.159678 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329329159663, "job": 1143, "event": "table_file_deletion", "file_number": 10839}
>2018-07-11 17:15:29.159707 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:16:05.734744 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:16:19.478712 7f29149e4700  1 mon.magna102@0(leader).log v21798 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21798)
>2018-07-11 17:17:05.756745 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:18:05.786955 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:19:05.808821 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:19:59.472075 7f29149e4700  1 mon.magna102@0(leader).log v21799 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21799)
>2018-07-11 17:20:05.839513 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:21:05.852246 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:22:05.890813 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:22:43.995038 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10843. Immutable memtables: 0.
>
>2018-07-11 17:22:43.995081 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1144] Syncing log #10840
>2018-07-11 17:22:44.011583 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:22:43.995072) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:22:44.011595 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1144] Flushing memtable with next log file: 10843
>
>2018-07-11 17:22:44.011609 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764011601, "job": 1144, "event": "flush_started", "num_memtables": 1, "num_entries": 1805, "num_deletes": 251, "memory_usage": 3306744}
>2018-07-11 17:22:44.011612 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1144] Level-0 flush table #10844: started
>2018-07-11 17:22:44.078811 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764078796, "cf_name": "default", "job": 1144, "event": "table_file_creation", "file_number": 10844, "file_size": 3155441, "table_properties": {"data_size": 3129127, "index_size": 14658, "filter_size": 10732, "raw_key_size": 14615, "raw_average_key_size": 20, "raw_value_size": 3108772, "raw_average_value_size": 4403, "num_data_blocks": 442, "num_entries": 706, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:22:44.078835 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1144] Level-0 flush table #10844: 3155441 bytes OK
>2018-07-11 17:22:44.128826 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.095289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10844 started
>2018-07-11 17:22:44.128835 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.128788) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10844: memtable #1 done
>2018-07-11 17:22:44.128837 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.128802) EVENT_LOG_v1 {"time_micros": 1531329764128796, "job": 1144, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:22:44.128852 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.128816) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:22:44.128858 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1144] Try to delete WAL files size 3270295, prev total WAL file size 3270295, number of live WAL files 2.
>
>2018-07-11 17:22:44.129736 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:22:44.129827 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.129818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:22:44.129839 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1145] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:22:44.129847 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1144 Base level 0, inputs: [10844(3081KB)], [10842(13MB)]
>
>2018-07-11 17:22:44.129861 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764129851, "job": 1145, "event": "compaction_started", "files_L0": [10844], "files_L1": [10842], "score": -1, "input_data_size": 17089138}
>2018-07-11 17:22:44.312851 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1145] Generated table #10845: 3042 keys, 15116702 bytes
>2018-07-11 17:22:44.312896 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764312869, "cf_name": "default", "job": 1145, "event": "table_file_creation", "file_number": 10845, "file_size": 15116702, "table_properties": {"data_size": 14999022, "index_size": 67866, "filter_size": 48887, "raw_key_size": 62682, "raw_average_key_size": 20, "raw_value_size": 14907776, "raw_average_value_size": 4900, "num_data_blocks": 1971, "num_entries": 3042, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:22:44.346019 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1145] Compacted 1@0 + 1@1 files to L1 => 15116702 bytes
>2018-07-11 17:22:44.379557 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.379492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 69.9 wr, level 1, files in(1, 1) out(1) MB in(3.0, 13.3) out(14.4), read-write-amplify(10.2) write-amplify(4.8) OK, records in: 3554, records dropped: 512
>
>2018-07-11 17:22:44.379565 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:22:44.379513) EVENT_LOG_v1 {"time_micros": 1531329764379503, "job": 1145, "event": "compaction_finished", "compaction_time_micros": 216144, "output_level": 1, "num_output_files": 1, "total_output_size": 15116702, "num_input_records": 3554, "num_output_records": 3042, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:22:44.380266 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764380262, "job": 1145, "event": "table_file_deletion", "file_number": 10844}
>2018-07-11 17:22:44.382996 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329764382993, "job": 1145, "event": "table_file_deletion", "file_number": 10842}
>2018-07-11 17:23:05.906293 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:23:49.010831 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10846. Immutable memtables: 0.
>
>2018-07-11 17:23:49.010873 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1146] Syncing log #10843
>2018-07-11 17:23:49.027401 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.010864) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:23:49.027412 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1146] Flushing memtable with next log file: 10846
>
>2018-07-11 17:23:49.027438 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829027430, "job": 1146, "event": "flush_started", "num_memtables": 1, "num_entries": 479, "num_deletes": 250, "memory_usage": 496960}
>2018-07-11 17:23:49.027441 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1146] Level-0 flush table #10847: started
>2018-07-11 17:23:49.069576 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829069563, "cf_name": "default", "job": 1146, "event": "table_file_creation", "file_number": 10847, "file_size": 294163, "table_properties": {"data_size": 290377, "index_size": 1305, "filter_size": 1562, "raw_key_size": 6450, "raw_average_key_size": 21, "raw_value_size": 285452, "raw_average_value_size": 967, "num_data_blocks": 41, "num_entries": 295, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:23:49.069586 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1146] Level-0 flush table #10847: 294163 bytes OK
>2018-07-11 17:23:49.119611 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.086107) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10847 started
>2018-07-11 17:23:49.119619 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.119576) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10847: memtable #1 done
>2018-07-11 17:23:49.119635 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.119591) EVENT_LOG_v1 {"time_micros": 1531329829119584, "job": 1146, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:23:49.119637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.119602) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:23:49.119643 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1146] Try to delete WAL files size 485939, prev total WAL file size 485939, number of live WAL files 2.
>
>2018-07-11 17:23:49.119822 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:23:49.119874 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.119867) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:23:49.119883 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1147] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:23:49.119890 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1146 Base level 0, inputs: [10847(287KB)], [10845(14MB)]
>
>2018-07-11 17:23:49.119903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829119893, "job": 1147, "event": "compaction_started", "files_L0": [10847], "files_L1": [10845], "score": -1, "input_data_size": 15410865}
>2018-07-11 17:23:49.286981 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1147] Generated table #10848: 2857 keys, 13886944 bytes
>2018-07-11 17:23:49.287023 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829286997, "cf_name": "default", "job": 1147, "event": "table_file_creation", "file_number": 10848, "file_size": 13886944, "table_properties": {"data_size": 13779830, "index_size": 61345, "filter_size": 44842, "raw_key_size": 58546, "raw_average_key_size": 20, "raw_value_size": 13695865, "raw_average_value_size": 4793, "num_data_blocks": 1786, "num_entries": 2857, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:23:49.320168 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1147] Compacted 1@0 + 1@1 files to L1 => 13886944 bytes
>2018-07-11 17:23:49.353760 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.353682) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.0 rd, 69.3 wr, level 1, files in(1, 1) out(1) MB in(0.3, 14.4) out(13.2), read-write-amplify(99.6) write-amplify(47.2) OK, records in: 3337, records dropped: 480
>
>2018-07-11 17:23:49.353770 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:23:49.353706) EVENT_LOG_v1 {"time_micros": 1531329829353696, "job": 1147, "event": "compaction_finished", "compaction_time_micros": 200249, "output_level": 1, "num_output_files": 1, "total_output_size": 13886944, "num_input_records": 3337, "num_output_records": 2857, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:23:49.353903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829353898, "job": 1147, "event": "table_file_deletion", "file_number": 10847}
>2018-07-11 17:23:49.356708 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531329829356705, "job": 1147, "event": "table_file_deletion", "file_number": 10845}
>2018-07-11 17:23:49.356822 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:24:05.938517 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:25:05.960463 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:26:05.990620 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:27:06.012500 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:28:06.041355 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:29:06.063295 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:30:06.093535 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:31:02.224298 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10849. Immutable memtables: 0.
>
>2018-07-11 17:31:02.224344 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1148] Syncing log #10846
>2018-07-11 17:31:02.240854 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.224334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:31:02.240866 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1148] Flushing memtable with next log file: 10849
>
>2018-07-11 17:31:02.240892 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262240884, "job": 1148, "event": "flush_started", "num_memtables": 1, "num_entries": 1775, "num_deletes": 251, "memory_usage": 3252888}
>2018-07-11 17:31:02.240895 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1148] Level-0 flush table #10850: started
>2018-07-11 17:31:02.308049 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262308035, "cf_name": "default", "job": 1148, "event": "table_file_creation", "file_number": 10850, "file_size": 3102890, "table_properties": {"data_size": 3076941, "index_size": 14457, "filter_size": 10568, "raw_key_size": 14310, "raw_average_key_size": 20, "raw_value_size": 3056987, "raw_average_value_size": 4417, "num_data_blocks": 436, "num_entries": 692, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:31:02.308073 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1148] Level-0 flush table #10850: 3102890 bytes OK
>2018-07-11 17:31:02.358060 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.324582) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10850 started
>2018-07-11 17:31:02.358068 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.358022) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10850: memtable #1 done
>2018-07-11 17:31:02.358070 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.358036) EVENT_LOG_v1 {"time_micros": 1531330262358030, "job": 1148, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:31:02.358072 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.358050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:31:02.358092 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1148] Try to delete WAL files size 3217194, prev total WAL file size 3217194, number of live WAL files 2.
>
>2018-07-11 17:31:02.358912 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:31:02.359132 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 17:31:02.359141 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 92289.2 total, 990.4 interval
>Cumulative writes: 118K writes, 556K keys, 118K commit groups, 1.0 writes per commit group, ingest: 0.91 GB, 0.01 MB/s
>Cumulative WAL: 118K writes, 118K syncs, 1.00 writes per sync, written: 0.91 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4510 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.06 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.96 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       575    0.079       0      0
>  L1      1/1   13.24 MB   0.0      9.1     0.8      8.3       8.3     -0.0       0.0  10.8     72.6     66.5       128       572    0.223   2056K   248K
> Sum      2/2   16.20 MB   0.0      9.1     0.8      8.3       9.1      0.8       0.0  11.7     53.6     53.6       173      1147    0.151   2056K   248K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.5     54.4     54.2         1         8    0.143     13K   1986
>Uptime(secs): 92289.2 total, 92289.2 interval
>Flush(GB): cumulative 0.772, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.07 GB write, 0.10 MB/s write, 9.07 GB read, 0.10 MB/s read, 173.2 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.96 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        45       575    0.079       0      0
>  L1      1/1   13.24 MB   0.0      9.1     0.8      8.3       8.3     -0.0       0.0  10.8     72.6     66.5       128       572    0.223   2056K   248K
> Sum      2/2   16.20 MB   0.0      9.1     0.8      8.3       9.1      0.8       0.0  11.7     53.6     53.6       173      1147    0.151   2056K   248K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 92289.2 total, 92289.2 interval
>Flush(GB): cumulative 0.772, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.07 GB write, 0.10 MB/s write, 9.07 GB read, 0.10 MB/s read, 173.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 17:31:02.359164 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.359157) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:31:02.359171 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1149] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:31:02.359176 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1148 Base level 0, inputs: [10850(3030KB)], [10848(13MB)]
>
>2018-07-11 17:31:02.359210 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262359200, "job": 1149, "event": "compaction_started", "files_L0": [10850], "files_L1": [10848], "score": -1, "input_data_size": 16989834}
>2018-07-11 17:31:02.542100 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1149] Generated table #10851: 3041 keys, 15086846 bytes
>2018-07-11 17:31:02.542150 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262542122, "cf_name": "default", "job": 1149, "event": "table_file_creation", "file_number": 10851, "file_size": 15086846, "table_properties": {"data_size": 14969266, "index_size": 67831, "filter_size": 48822, "raw_key_size": 62660, "raw_average_key_size": 20, "raw_value_size": 14878059, "raw_average_value_size": 4892, "num_data_blocks": 1970, "num_entries": 3041, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:31:02.575279 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1149] Compacted 1@0 + 1@1 files to L1 => 15086846 bytes
>2018-07-11 17:31:02.608798 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.608731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.6 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(3.0, 13.2) out(14.4), read-write-amplify(10.3) write-amplify(4.9) OK, records in: 3549, records dropped: 508
>
>2018-07-11 17:31:02.608807 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:31:02.608753) EVENT_LOG_v1 {"time_micros": 1531330262608743, "job": 1149, "event": "compaction_finished", "compaction_time_micros": 216052, "output_level": 1, "num_output_files": 1, "total_output_size": 15086846, "num_input_records": 3549, "num_output_records": 3041, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:31:02.609510 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262609506, "job": 1149, "event": "table_file_deletion", "file_number": 10850}
>2018-07-11 17:31:02.612271 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330262612268, "job": 1149, "event": "table_file_deletion", "file_number": 10848}
>2018-07-11 17:31:06.117729 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:32:06.139572 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:32:09.206897 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10852. Immutable memtables: 0.
>
>2018-07-11 17:32:09.206937 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1150] Syncing log #10849
>2018-07-11 17:32:09.223443 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.206928) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:32:09.223454 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1150] Flushing memtable with next log file: 10852
>
>2018-07-11 17:32:09.223480 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329223472, "job": 1150, "event": "flush_started", "num_memtables": 1, "num_entries": 486, "num_deletes": 250, "memory_usage": 511872}
>2018-07-11 17:32:09.223484 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1150] Level-0 flush table #10853: started
>2018-07-11 17:32:09.265621 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329265607, "cf_name": "default", "job": 1150, "event": "table_file_creation", "file_number": 10853, "file_size": 301619, "table_properties": {"data_size": 297779, "index_size": 1334, "filter_size": 1587, "raw_key_size": 6470, "raw_average_key_size": 21, "raw_value_size": 292815, "raw_average_value_size": 989, "num_data_blocks": 42, "num_entries": 296, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:32:09.265631 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1150] Level-0 flush table #10853: 301619 bytes OK
>2018-07-11 17:32:09.315668 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.282175) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10853 started
>2018-07-11 17:32:09.315676 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.315633) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10853: memtable #1 done
>2018-07-11 17:32:09.315679 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.315648) EVENT_LOG_v1 {"time_micros": 1531330329315641, "job": 1150, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:32:09.315681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.315659) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:32:09.315699 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1150] Try to delete WAL files size 500708, prev total WAL file size 500708, number of live WAL files 2.
>
>2018-07-11 17:32:09.315880 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:32:09.315928 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.315921) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:32:09.315937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1151] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:32:09.315944 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1150 Base level 0, inputs: [10853(294KB)], [10851(14MB)]
>
>2018-07-11 17:32:09.315957 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329315947, "job": 1151, "event": "compaction_started", "files_L0": [10853], "files_L1": [10851], "score": -1, "input_data_size": 15388465}
>2018-07-11 17:32:09.491345 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1151] Generated table #10854: 2858 keys, 13871340 bytes
>2018-07-11 17:32:09.491392 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329491364, "cf_name": "default", "job": 1151, "event": "table_file_creation", "file_number": 10854, "file_size": 13871340, "table_properties": {"data_size": 13764219, "index_size": 61375, "filter_size": 44819, "raw_key_size": 58566, "raw_average_key_size": 20, "raw_value_size": 13680217, "raw_average_value_size": 4786, "num_data_blocks": 1787, "num_entries": 2858, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:32:09.524508 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1151] Compacted 1@0 + 1@1 files to L1 => 13871340 bytes
>2018-07-11 17:32:09.558099 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.558024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.8 rd, 66.5 wr, level 1, files in(1, 1) out(1) MB in(0.3, 14.4) out(13.2), read-write-amplify(97.0) write-amplify(46.0) OK, records in: 3337, records dropped: 479
>
>2018-07-11 17:32:09.558109 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:32:09.558049) EVENT_LOG_v1 {"time_micros": 1531330329558038, "job": 1151, "event": "compaction_finished", "compaction_time_micros": 208534, "output_level": 1, "num_output_files": 1, "total_output_size": 13871340, "num_input_records": 3337, "num_output_records": 2858, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:32:09.558266 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329558262, "job": 1151, "event": "table_file_deletion", "file_number": 10853}
>2018-07-11 17:32:09.561017 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330329561015, "job": 1151, "event": "table_file_deletion", "file_number": 10851}
>2018-07-11 17:32:09.561132 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:33:06.163528 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:34:06.185344 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:35:06.215557 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:35:41.334708 7f29149e4700  1 mon.magna102@0(leader).log v21800 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21800)
>2018-07-11 17:36:06.244520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:37:06.266503 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:38:06.296681 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:38:07.421933 7f29149e4700  1 mon.magna102@0(leader).log v21801 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21801)
>2018-07-11 17:39:06.318958 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:39:16.420159 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10855. Immutable memtables: 0.
>
>2018-07-11 17:39:16.420252 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1152] Syncing log #10852
>2018-07-11 17:39:16.436637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.420243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:39:16.436648 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1152] Flushing memtable with next log file: 10855
>
>2018-07-11 17:39:16.436675 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756436667, "job": 1152, "event": "flush_started", "num_memtables": 1, "num_entries": 1770, "num_deletes": 251, "memory_usage": 3243728}
>2018-07-11 17:39:16.436679 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1152] Level-0 flush table #10856: started
>2018-07-11 17:39:16.503935 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756503920, "cf_name": "default", "job": 1152, "event": "table_file_creation", "file_number": 10856, "file_size": 3095645, "table_properties": {"data_size": 3069799, "index_size": 14386, "filter_size": 10536, "raw_key_size": 14357, "raw_average_key_size": 20, "raw_value_size": 3049832, "raw_average_value_size": 4394, "num_data_blocks": 434, "num_entries": 694, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:39:16.503946 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1152] Level-0 flush table #10856: 3095645 bytes OK
>2018-07-11 17:39:16.553951 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.520460) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10856 started
>2018-07-11 17:39:16.553959 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.553912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10856: memtable #1 done
>2018-07-11 17:39:16.553961 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.553927) EVENT_LOG_v1 {"time_micros": 1531330756553921, "job": 1152, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:39:16.553976 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.553940) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:39:16.553983 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1152] Try to delete WAL files size 3208279, prev total WAL file size 3208279, number of live WAL files 2.
>
>2018-07-11 17:39:16.554811 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:39:16.554899 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.554891) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:39:16.554912 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1153] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:39:16.554919 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1152 Base level 0, inputs: [10856(3023KB)], [10854(13MB)]
>
>2018-07-11 17:39:16.554934 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756554923, "job": 1153, "event": "compaction_started", "files_L0": [10856], "files_L1": [10854], "score": -1, "input_data_size": 16966985}
>2018-07-11 17:39:16.737993 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1153] Generated table #10857: 3042 keys, 15090041 bytes
>2018-07-11 17:39:16.738041 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756738013, "cf_name": "default", "job": 1153, "event": "table_file_creation", "file_number": 10857, "file_size": 15090041, "table_properties": {"data_size": 14972455, "index_size": 67828, "filter_size": 48831, "raw_key_size": 62676, "raw_average_key_size": 20, "raw_value_size": 14881219, "raw_average_value_size": 4891, "num_data_blocks": 1970, "num_entries": 3042, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:39:16.771138 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1153] Compacted 1@0 + 1@1 files to L1 => 15090041 bytes
>2018-07-11 17:39:16.804702 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.804612) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.5 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(3.0, 13.2) out(14.4), read-write-amplify(10.4) write-amplify(4.9) OK, records in: 3552, records dropped: 510
>
>2018-07-11 17:39:16.804711 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:39:16.804634) EVENT_LOG_v1 {"time_micros": 1531330756804624, "job": 1153, "event": "compaction_finished", "compaction_time_micros": 216187, "output_level": 1, "num_output_files": 1, "total_output_size": 15090041, "num_input_records": 3552, "num_output_records": 3042, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:39:16.805416 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756805412, "job": 1153, "event": "table_file_deletion", "file_number": 10856}
>2018-07-11 17:39:16.808123 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330756808120, "job": 1153, "event": "table_file_deletion", "file_number": 10854}
>2018-07-11 17:40:06.334765 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:40:29.436654 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10858. Immutable memtables: 0.
>
>2018-07-11 17:40:29.436697 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1154] Syncing log #10855
>2018-07-11 17:40:29.453214 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.436687) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:40:29.453226 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1154] Flushing memtable with next log file: 10858
>
>2018-07-11 17:40:29.453252 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829453244, "job": 1154, "event": "flush_started", "num_memtables": 1, "num_entries": 507, "num_deletes": 250, "memory_usage": 556632}
>2018-07-11 17:40:29.453255 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1154] Level-0 flush table #10859: started
>2018-07-11 17:40:29.495423 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829495408, "cf_name": "default", "job": 1154, "event": "table_file_creation", "file_number": 10859, "file_size": 323986, "table_properties": {"data_size": 319977, "index_size": 1432, "filter_size": 1658, "raw_key_size": 6530, "raw_average_key_size": 21, "raw_value_size": 314904, "raw_average_value_size": 1053, "num_data_blocks": 45, "num_entries": 299, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:40:29.495433 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1154] Level-0 flush table #10859: 323986 bytes OK
>2018-07-11 17:40:29.553783 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.511940) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10859 started
>2018-07-11 17:40:29.553791 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.553747) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10859: memtable #1 done
>2018-07-11 17:40:29.553794 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.553762) EVENT_LOG_v1 {"time_micros": 1531330829553755, "job": 1154, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:40:29.553795 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.553773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:40:29.553813 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1154] Try to delete WAL files size 545015, prev total WAL file size 545015, number of live WAL files 2.
>
>2018-07-11 17:40:29.554020 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:40:29.554069 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.554061) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:40:29.554078 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1155] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:40:29.554084 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1154 Base level 0, inputs: [10859(316KB)], [10857(14MB)]
>
>2018-07-11 17:40:29.554097 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829554088, "job": 1155, "event": "compaction_started", "files_L0": [10859], "files_L1": [10857], "score": -1, "input_data_size": 15414027}
>2018-07-11 17:40:29.737806 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1155] Generated table #10860: 2865 keys, 13917202 bytes
>2018-07-11 17:40:29.737854 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829737826, "cf_name": "default", "job": 1155, "event": "table_file_creation", "file_number": 10860, "file_size": 13917202, "table_properties": {"data_size": 13809731, "index_size": 61578, "filter_size": 44966, "raw_key_size": 58708, "raw_average_key_size": 20, "raw_value_size": 13725472, "raw_average_value_size": 4790, "num_data_blocks": 1793, "num_entries": 2865, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:40:29.771000 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1155] Compacted 1@0 + 1@1 files to L1 => 13917202 bytes
>2018-07-11 17:40:29.812958 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.812882) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 71.1 rd, 64.2 wr, level 1, files in(1, 1) out(1) MB in(0.3, 14.4) out(13.3), read-write-amplify(90.5) write-amplify(43.0) OK, records in: 3341, records dropped: 476
>
>2018-07-11 17:40:29.812969 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:40:29.812907) EVENT_LOG_v1 {"time_micros": 1531330829812897, "job": 1155, "event": "compaction_finished", "compaction_time_micros": 216885, "output_level": 1, "num_output_files": 1, "total_output_size": 13917202, "num_input_records": 3341, "num_output_records": 2865, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:40:29.813110 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829813105, "job": 1155, "event": "table_file_deletion", "file_number": 10859}
>2018-07-11 17:40:29.815899 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531330829815896, "job": 1155, "event": "table_file_deletion", "file_number": 10857}
>2018-07-11 17:40:29.816012 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:41:06.334899 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:42:06.335078 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:43:06.335250 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:44:06.335437 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:45:06.335610 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:46:01.599774 7f29149e4700  1 mon.magna102@0(leader).log v21802 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21802)
>2018-07-11 17:46:06.335788 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:46:16.469969 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 17:46:16.626107 7f29149e4700  1 mon.magna102@0(leader).log v21803 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21803)
>2018-07-11 17:46:24.509755 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:46:24.626675 7f29149e4700  1 mon.magna102@0(leader).log v21804 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21804)
>2018-07-11 17:47:06.335963 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:47:26.501154 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:47:26.649096 7f29149e4700  1 mon.magna102@0(leader).log v21805 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21805)
>2018-07-11 17:47:26.716146 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10861. Immutable memtables: 0.
>
>2018-07-11 17:47:26.716195 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1156] Syncing log #10858
>2018-07-11 17:47:26.732693 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.716185) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:47:26.732704 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1156] Flushing memtable with next log file: 10861
>
>2018-07-11 17:47:26.732730 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331246732722, "job": 1156, "event": "flush_started", "num_memtables": 1, "num_entries": 1751, "num_deletes": 251, "memory_usage": 3221320}
>2018-07-11 17:47:26.732734 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1156] Level-0 flush table #10862: started
>2018-07-11 17:47:26.799889 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331246799874, "cf_name": "default", "job": 1156, "event": "table_file_creation", "file_number": 10862, "file_size": 3070485, "table_properties": {"data_size": 3044935, "index_size": 14187, "filter_size": 10439, "raw_key_size": 14269, "raw_average_key_size": 20, "raw_value_size": 3025173, "raw_average_value_size": 4384, "num_data_blocks": 428, "num_entries": 690, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:47:26.799900 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1156] Level-0 flush table #10862: 3070485 bytes OK
>2018-07-11 17:47:26.849904 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.816412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10862 started
>2018-07-11 17:47:26.849912 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.849866) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10862: memtable #1 done
>2018-07-11 17:47:26.849915 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.849881) EVENT_LOG_v1 {"time_micros": 1531331246849875, "job": 1156, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:47:26.849930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.849894) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:47:26.849937 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1156] Try to delete WAL files size 3186419, prev total WAL file size 3186419, number of live WAL files 2.
>
>2018-07-11 17:47:26.850778 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:47:26.851000 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 17:47:26.851009 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 93273.7 total, 984.5 interval
>Cumulative writes: 119K writes, 560K keys, 119K commit groups, 1.0 writes per commit group, ingest: 0.92 GB, 0.01 MB/s
>Cumulative WAL: 119K writes, 119K syncs, 1.00 writes per sync, written: 0.92 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4514 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.10 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.93 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        46       579    0.079       0      0
>  L1      1/1   13.27 MB   0.0      9.1     0.8      8.3       8.3     -0.0       0.0  10.8     72.6     66.4       129       576    0.223   2070K   250K
> Sum      2/2   16.20 MB   0.0      9.1     0.8      8.3       9.1      0.8       0.0  11.7     53.6     53.6       174      1155    0.151   2070K   250K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.5     54.0     54.0         1         8    0.143     13K   1973
>Uptime(secs): 93273.7 total, 93273.7 interval
>Flush(GB): cumulative 0.779, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.13 GB write, 0.10 MB/s write, 9.13 GB read, 0.10 MB/s read, 174.4 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.93 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.4        46       579    0.079       0      0
>  L1      1/1   13.27 MB   0.0      9.1     0.8      8.3       8.3     -0.0       0.0  10.8     72.6     66.4       129       576    0.223   2070K   250K
> Sum      2/2   16.20 MB   0.0      9.1     0.8      8.3       9.1      0.8       0.0  11.7     53.6     53.6       174      1155    0.151   2070K   250K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 93273.7 total, 93273.7 interval
>Flush(GB): cumulative 0.779, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.13 GB write, 0.10 MB/s write, 9.13 GB read, 0.10 MB/s read, 174.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 17:47:26.851031 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:47:26.851024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:47:26.851036 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1157] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:47:26.851042 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1156 Base level 0, inputs: [10862(2998KB)], [10860(13MB)]
>
>2018-07-11 17:47:26.851055 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331246851045, "job": 1157, "event": "compaction_started", "files_L0": [10862], "files_L1": [10860], "score": -1, "input_data_size": 16987687}
>2018-07-11 17:47:27.042272 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1157] Generated table #10863: 3045 keys, 15107739 bytes
>2018-07-11 17:47:27.042314 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331247042299, "cf_name": "default", "job": 1157, "event": "table_file_creation", "file_number": 10863, "file_size": 15107739, "table_properties": {"data_size": 14990115, "index_size": 67828, "filter_size": 48869, "raw_key_size": 62730, "raw_average_key_size": 20, "raw_value_size": 14898847, "raw_average_value_size": 4892, "num_data_blocks": 1970, "num_entries": 3045, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:47:27.075459 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1157] Compacted 1@0 + 1@1 files to L1 => 15107739 bytes
>2018-07-11 17:47:27.108982 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:47:27.108913) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.7 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(2.9, 13.3) out(14.4), read-write-amplify(10.5) write-amplify(4.9) OK, records in: 3555, records dropped: 510
>
>2018-07-11 17:47:27.108991 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:47:27.108934) EVENT_LOG_v1 {"time_micros": 1531331247108925, "job": 1157, "event": "compaction_finished", "compaction_time_micros": 224386, "output_level": 1, "num_output_files": 1, "total_output_size": 15107739, "num_input_records": 3555, "num_output_records": 3045, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:47:27.109682 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331247109678, "job": 1157, "event": "table_file_deletion", "file_number": 10862}
>2018-07-11 17:47:27.112465 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331247112462, "job": 1157, "event": "table_file_deletion", "file_number": 10860}
>2018-07-11 17:47:40.507535 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:47:40.660658 7f29149e4700  1 mon.magna102@0(leader).log v21806 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21806)
>2018-07-11 17:47:56.514464 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 17:47:56.671429 7f29149e4700  1 mon.magna102@0(leader).log v21807 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21807)
>2018-07-11 17:48:04.555149 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:48:04.672150 7f29149e4700  1 mon.magna102@0(leader).log v21808 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21808)
>2018-07-11 17:48:06.336149 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:48:49.650571 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10864. Immutable memtables: 0.
>
>2018-07-11 17:48:49.650615 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1158] Syncing log #10861
>2018-07-11 17:48:49.667125 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.650605) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:48:49.667136 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1158] Flushing memtable with next log file: 10864
>
>2018-07-11 17:48:49.667162 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329667154, "job": 1158, "event": "flush_started", "num_memtables": 1, "num_entries": 566, "num_deletes": 250, "memory_usage": 701112}
>2018-07-11 17:48:49.667165 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1158] Level-0 flush table #10865: started
>2018-07-11 17:48:49.709313 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329709299, "cf_name": "default", "job": 1158, "event": "table_file_creation", "file_number": 10865, "file_size": 422974, "table_properties": {"data_size": 418350, "index_size": 1755, "filter_size": 1950, "raw_key_size": 6864, "raw_average_key_size": 21, "raw_value_size": 412794, "raw_average_value_size": 1310, "num_data_blocks": 55, "num_entries": 315, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:48:49.709324 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1158] Level-0 flush table #10865: 422974 bytes OK
>2018-07-11 17:48:49.760566 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.727075) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10865 started
>2018-07-11 17:48:49.760574 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.760530) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10865: memtable #1 done
>2018-07-11 17:48:49.760576 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.760545) EVENT_LOG_v1 {"time_micros": 1531331329760539, "job": 1158, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:48:49.760578 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.760556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:48:49.760598 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1158] Try to delete WAL files size 688720, prev total WAL file size 688720, number of live WAL files 2.
>
>2018-07-11 17:48:49.760888 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:48:49.760938 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.760931) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:48:49.760948 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1159] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:48:49.760954 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1158 Base level 0, inputs: [10865(413KB)], [10863(14MB)]
>
>2018-07-11 17:48:49.760967 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329760958, "job": 1159, "event": "compaction_started", "files_L0": [10865], "files_L1": [10863], "score": -1, "input_data_size": 15530713}
>2018-07-11 17:48:49.927939 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1159] Generated table #10866: 2887 keys, 14064397 bytes
>2018-07-11 17:48:49.927988 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329927960, "cf_name": "default", "job": 1159, "event": "table_file_creation", "file_number": 10866, "file_size": 14064397, "table_properties": {"data_size": 13955940, "index_size": 62123, "filter_size": 45407, "raw_key_size": 59155, "raw_average_key_size": 20, "raw_value_size": 13870999, "raw_average_value_size": 4804, "num_data_blocks": 1809, "num_entries": 2887, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:48:49.961120 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1159] Compacted 1@0 + 1@1 files to L1 => 14064397 bytes
>2018-07-11 17:48:49.994708 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.994632) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 77.6 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.4, 14.4) out(13.4), read-write-amplify(70.0) write-amplify(33.3) OK, records in: 3360, records dropped: 473
>
>2018-07-11 17:48:49.994718 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:48:49.994657) EVENT_LOG_v1 {"time_micros": 1531331329994647, "job": 1159, "event": "compaction_finished", "compaction_time_micros": 200134, "output_level": 1, "num_output_files": 1, "total_output_size": 14064397, "num_input_records": 3360, "num_output_records": 2887, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:48:49.994881 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329994877, "job": 1159, "event": "table_file_deletion", "file_number": 10865}
>2018-07-11 17:48:49.997687 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331329997684, "job": 1159, "event": "table_file_deletion", "file_number": 10863}
>2018-07-11 17:48:49.997804 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:49:06.336336 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:49:06.546122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:49:06.704916 7f29149e4700  1 mon.magna102@0(leader).log v21809 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21809)
>2018-07-11 17:49:20.552413 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:49:20.722666 7f29149e4700  1 mon.magna102@0(leader).log v21810 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21810)
>2018-07-11 17:49:40.561149 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:49:40.707563 7f29149e4700  1 mon.magna102@0(leader).log v21811 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21811)
>2018-07-11 17:50:06.336518 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:50:50.592172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:50:50.747562 7f29149e4700  1 mon.magna102@0(leader).log v21812 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21812)
>2018-07-11 17:51:00.596539 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:51:00.766287 7f29149e4700  1 mon.magna102@0(leader).log v21813 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21813)
>2018-07-11 17:51:06.336699 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:51:20.605264 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:51:20.776167 7f29149e4700  1 mon.magna102@0(leader).log v21814 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21814)
>2018-07-11 17:52:06.336889 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:52:30.635974 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:52:30.790433 7f29149e4700  1 mon.magna102@0(leader).log v21815 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21815)
>2018-07-11 17:52:44.641957 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:52:44.808223 7f29149e4700  1 mon.magna102@0(leader).log v21816 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21816)
>2018-07-11 17:53:00.649105 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:53:00.817791 7f29149e4700  1 mon.magna102@0(leader).log v21817 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21817)
>2018-07-11 17:53:06.337016 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:54:06.337200 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:54:16.682397 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:54:16.842107 7f29149e4700  1 mon.magna102@0(leader).log v21818 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21818)
>2018-07-11 17:54:24.685839 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:54:24.842846 7f29149e4700  1 mon.magna102@0(leader).log v21819 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21819)
>2018-07-11 17:54:40.692898 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:54:40.860784 7f29149e4700  1 mon.magna102@0(leader).log v21820 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21820)
>2018-07-11 17:55:06.337385 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:55:14.838534 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10867. Immutable memtables: 0.
>
>2018-07-11 17:55:14.838577 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1160] Syncing log #10864
>2018-07-11 17:55:14.856304 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.838567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:55:14.856315 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1160] Flushing memtable with next log file: 10867
>
>2018-07-11 17:55:14.856341 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331714856333, "job": 1160, "event": "flush_started", "num_memtables": 1, "num_entries": 1703, "num_deletes": 251, "memory_usage": 3188264}
>2018-07-11 17:55:14.856344 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1160] Level-0 flush table #10868: started
>2018-07-11 17:55:14.915232 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331714915218, "cf_name": "default", "job": 1160, "event": "table_file_creation", "file_number": 10868, "file_size": 3026578, "table_properties": {"data_size": 3001763, "index_size": 13660, "filter_size": 10231, "raw_key_size": 14085, "raw_average_key_size": 20, "raw_value_size": 2982490, "raw_average_value_size": 4373, "num_data_blocks": 412, "num_entries": 682, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 17:55:14.915243 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1160] Level-0 flush table #10868: 3026578 bytes OK
>2018-07-11 17:55:14.973565 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.931744) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10868 started
>2018-07-11 17:55:14.973572 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.973526) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10868: memtable #1 done
>2018-07-11 17:55:14.973575 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.973540) EVENT_LOG_v1 {"time_micros": 1531331714973534, "job": 1160, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:55:14.973589 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.973554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:55:14.973596 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1160] Try to delete WAL files size 3154890, prev total WAL file size 3154890, number of live WAL files 2.
>
>2018-07-11 17:55:14.974391 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:55:14.974471 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:55:14.974462) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 17:55:14.974483 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1161] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:55:14.974491 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1160 Base level 0, inputs: [10868(2955KB)], [10866(13MB)]
>
>2018-07-11 17:55:14.974505 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331714974495, "job": 1161, "event": "compaction_started", "files_L0": [10868], "files_L1": [10866], "score": -1, "input_data_size": 17090975}
>2018-07-11 17:55:15.157619 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1161] Generated table #10869: 3059 keys, 15207857 bytes
>2018-07-11 17:55:15.157663 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331715157636, "cf_name": "default", "job": 1161, "event": "table_file_creation", "file_number": 10869, "file_size": 15207857, "table_properties": {"data_size": 15090027, "index_size": 67810, "filter_size": 49093, "raw_key_size": 62993, "raw_average_key_size": 20, "raw_value_size": 14998574, "raw_average_value_size": 4903, "num_data_blocks": 1969, "num_entries": 3059, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:55:15.190760 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1161] Compacted 1@0 + 1@1 files to L1 => 15207857 bytes
>2018-07-11 17:55:15.232660 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:55:15.232593) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.0 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(2.9, 13.4) out(14.5), read-write-amplify(10.7) write-amplify(5.0) OK, records in: 3569, records dropped: 510
>
>2018-07-11 17:55:15.232668 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:55:15.232614) EVENT_LOG_v1 {"time_micros": 1531331715232605, "job": 1161, "event": "compaction_finished", "compaction_time_micros": 216238, "output_level": 1, "num_output_files": 1, "total_output_size": 15207857, "num_input_records": 3569, "num_output_records": 3059, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:55:15.233346 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331715233342, "job": 1161, "event": "table_file_deletion", "file_number": 10868}
>2018-07-11 17:55:15.236119 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331715236116, "job": 1161, "event": "table_file_deletion", "file_number": 10866}
>2018-07-11 17:55:56.726555 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:55:56.877864 7f29149e4700  1 mon.magna102@0(leader).log v21821 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21821)
>2018-07-11 17:56:04.730032 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:56:04.878590 7f29149e4700  1 mon.magna102@0(leader).log v21822 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21822)
>2018-07-11 17:56:06.337573 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:56:20.737109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:56:20.897761 7f29149e4700  1 mon.magna102@0(leader).log v21823 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21823)
>2018-07-11 17:57:06.337761 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 17:57:09.851392 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10870. Immutable memtables: 0.
>
>2018-07-11 17:57:09.851435 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1162] Syncing log #10867
>2018-07-11 17:57:09.867889 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.851425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 17:57:09.867900 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1162] Flushing memtable with next log file: 10870
>
>2018-07-11 17:57:09.867926 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331829867918, "job": 1162, "event": "flush_started", "num_memtables": 1, "num_entries": 678, "num_deletes": 250, "memory_usage": 947272}
>2018-07-11 17:57:09.867929 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1162] Level-0 flush table #10871: started
>2018-07-11 17:57:09.910103 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331829910089, "cf_name": "default", "job": 1162, "event": "table_file_creation", "file_number": 10871, "file_size": 546260, "table_properties": {"data_size": 540745, "index_size": 2266, "filter_size": 2330, "raw_key_size": 7184, "raw_average_key_size": 21, "raw_value_size": 534599, "raw_average_value_size": 1615, "num_data_blocks": 71, "num_entries": 331, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 17:57:09.910113 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1162] Level-0 flush table #10871: 546260 bytes OK
>2018-07-11 17:57:09.968383 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.926550) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10871 started
>2018-07-11 17:57:09.968392 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.968345) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10871: memtable #1 done
>2018-07-11 17:57:09.968395 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.968360) EVENT_LOG_v1 {"time_micros": 1531331829968354, "job": 1162, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 17:57:09.968397 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.968372) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 17:57:09.968403 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1162] Try to delete WAL files size 932930, prev total WAL file size 932930, number of live WAL files 2.
>
>2018-07-11 17:57:09.968708 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:57:09.968762 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:57:09.968755) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 17:57:09.968771 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1163] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 17:57:09.968778 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1162 Base level 0, inputs: [10871(533KB)], [10869(14MB)]
>
>2018-07-11 17:57:09.968790 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331829968781, "job": 1163, "event": "compaction_started", "files_L0": [10871], "files_L1": [10869], "score": -1, "input_data_size": 15754117}
>2018-07-11 17:57:10.152503 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1163] Generated table #10872: 2933 keys, 14384655 bytes
>2018-07-11 17:57:10.152544 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331830152518, "cf_name": "default", "job": 1163, "event": "table_file_creation", "file_number": 10872, "file_size": 14384655, "table_properties": {"data_size": 14274163, "index_size": 63220, "filter_size": 46345, "raw_key_size": 60090, "raw_average_key_size": 20, "raw_value_size": 14187799, "raw_average_value_size": 4837, "num_data_blocks": 1841, "num_entries": 2933, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 17:57:10.194044 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1163] Compacted 1@0 + 1@1 files to L1 => 14384655 bytes
>2018-07-11 17:57:10.237106 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:57:10.237039) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.9 rd, 63.9 wr, level 1, files in(1, 1) out(1) MB in(0.5, 14.5) out(13.7), read-write-amplify(55.2) write-amplify(26.3) OK, records in: 3390, records dropped: 457
>
>2018-07-11 17:57:10.237115 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-17:57:10.237060) EVENT_LOG_v1 {"time_micros": 1531331830237051, "job": 1163, "event": "compaction_finished", "compaction_time_micros": 225238, "output_level": 1, "num_output_files": 1, "total_output_size": 14384655, "num_input_records": 3390, "num_output_records": 2933, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 17:57:10.237318 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331830237313, "job": 1163, "event": "table_file_deletion", "file_number": 10871}
>2018-07-11 17:57:10.240119 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531331830240117, "job": 1163, "event": "table_file_deletion", "file_number": 10869}
>2018-07-11 17:57:10.240238 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 17:57:30.768218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:57:30.922751 7f29149e4700  1 mon.magna102@0(leader).log v21824 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21824)
>2018-07-11 17:57:44.774257 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:57:44.923958 7f29149e4700  1 mon.magna102@0(leader).log v21825 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21825)
>2018-07-11 17:58:00.781231 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:58:00.966893 7f29149e4700  1 mon.magna102@0(leader).log v21826 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21826)
>2018-07-11 17:58:06.337939 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:59:06.338107 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 17:59:16.814704 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 17:59:16.965374 7f29149e4700  1 mon.magna102@0(leader).log v21827 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21827)
>2018-07-11 17:59:30.821220 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 17:59:30.976056 7f29149e4700  1 mon.magna102@0(leader).log v21828 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21828)
>2018-07-11 17:59:40.825500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 17:59:40.985361 7f29149e4700  1 mon.magna102@0(leader).log v21829 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21829)
>2018-07-11 17:59:49.868963 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 17:59:49.977809 7f29149e4700  1 mon.magna102@0(leader).log v21830 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21830)
>2018-07-11 18:00:06.338287 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:00:56.859503 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:00:57.009242 7f29149e4700  1 mon.magna102@0(leader).log v21831 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21831)
>2018-07-11 18:01:06.338477 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:01:10.866097 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:01:11.018981 7f29149e4700  1 mon.magna102@0(leader).log v21832 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21832)
>2018-07-11 18:01:26.873079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 18:01:27.028557 7f29149e4700  1 mon.magna102@0(leader).log v21833 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21833)
>2018-07-11 18:01:34.929144 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
>2018-07-11 18:01:35.055779 7f29149e4700  1 mon.magna102@0(leader).log v21834 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21834)
>2018-07-11 18:02:06.338647 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:02:36.903998 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:02:37.051803 7f29149e4700  1 mon.magna102@0(leader).log v21835 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21835)
>2018-07-11 18:02:50.910213 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:02:51.062502 7f29149e4700  1 mon.magna102@0(leader).log v21836 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21836)
>2018-07-11 18:03:01.046877 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10873. Immutable memtables: 0.
>
>2018-07-11 18:03:01.046921 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1164] Syncing log #10870
>2018-07-11 18:03:01.063448 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.046912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:03:01.063460 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1164] Flushing memtable with next log file: 10873
>
>2018-07-11 18:03:01.063474 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181063466, "job": 1164, "event": "flush_started", "num_memtables": 1, "num_entries": 1592, "num_deletes": 251, "memory_usage": 2951256}
>2018-07-11 18:03:01.063477 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1164] Level-0 flush table #10874: started
>2018-07-11 18:03:01.130670 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181130656, "cf_name": "default", "job": 1164, "event": "table_file_creation", "file_number": 10874, "file_size": 2802346, "table_properties": {"data_size": 2779328, "index_size": 12580, "filter_size": 9514, "raw_key_size": 13432, "raw_average_key_size": 20, "raw_value_size": 2761263, "raw_average_value_size": 4241, "num_data_blocks": 380, "num_entries": 651, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:03:01.130681 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1164] Level-0 flush table #10874: 2802346 bytes OK
>2018-07-11 18:03:01.180668 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.147188) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10874 started
>2018-07-11 18:03:01.180676 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.180630) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10874: memtable #1 done
>2018-07-11 18:03:01.180679 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.180645) EVENT_LOG_v1 {"time_micros": 1531332181180638, "job": 1164, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:03:01.180681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.180657) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:03:01.180687 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1164] Try to delete WAL files size 2920271, prev total WAL file size 2920271, number of live WAL files 2.
>
>2018-07-11 18:03:01.181394 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:03:01.181572 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 18:03:01.181578 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 94208.0 total, 934.3 interval
>Cumulative writes: 120K writes, 565K keys, 120K commit groups, 1.0 writes per commit group, ingest: 0.93 GB, 0.01 MB/s
>Cumulative WAL: 120K writes, 120K syncs, 1.00 writes per sync, written: 0.93 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4539 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.34 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.67 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       583    0.079       0      0
>  L1      1/1   13.72 MB   0.0      9.2     0.8      8.4       8.4     -0.0       0.0  10.7     72.6     66.4       130       580    0.223   2084K   252K
> Sum      2/2   16.39 MB   0.0      9.2     0.8      8.4       9.2      0.8       0.0  11.7     53.6     53.6       175      1163    0.151   2084K   252K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.6     54.5     54.7         1         8    0.143     13K   1950
>Uptime(secs): 94208.0 total, 94208.0 interval
>Flush(GB): cumulative 0.785, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.19 GB write, 0.10 MB/s write, 9.19 GB read, 0.10 MB/s read, 175.5 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.1 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.67 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       583    0.079       0      0
>  L1      1/1   13.72 MB   0.0      9.2     0.8      8.4       8.4     -0.0       0.0  10.7     72.6     66.4       130       580    0.223   2084K   252K
> Sum      2/2   16.39 MB   0.0      9.2     0.8      8.4       9.2      0.8       0.0  11.7     53.6     53.6       175      1163    0.151   2084K   252K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 94208.0 total, 94208.0 interval
>Flush(GB): cumulative 0.785, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.19 GB write, 0.10 MB/s write, 9.19 GB read, 0.10 MB/s read, 175.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 18:03:01.181599 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.181592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:03:01.181603 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1165] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:03:01.181608 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1164 Base level 0, inputs: [10874(2736KB)], [10872(13MB)]
>
>2018-07-11 18:03:01.181621 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181181611, "job": 1165, "event": "compaction_started", "files_L0": [10874], "files_L1": [10872], "score": -1, "input_data_size": 17187001}
>2018-07-11 18:03:01.364703 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1165] Generated table #10875: 3074 keys, 15292611 bytes
>2018-07-11 18:03:01.364747 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181364720, "cf_name": "default", "job": 1165, "event": "table_file_creation", "file_number": 10875, "file_size": 15292611, "table_properties": {"data_size": 15174573, "index_size": 67821, "filter_size": 49290, "raw_key_size": 63275, "raw_average_key_size": 20, "raw_value_size": 15082906, "raw_average_value_size": 4906, "num_data_blocks": 1969, "num_entries": 3074, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:03:01.397919 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1165] Compacted 1@0 + 1@1 files to L1 => 15292611 bytes
>2018-07-11 18:03:01.431443 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.431373) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(2.7, 13.7) out(14.6), read-write-amplify(11.6) write-amplify(5.5) OK, records in: 3584, records dropped: 510
>
>2018-07-11 18:03:01.431452 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:03:01.431395) EVENT_LOG_v1 {"time_micros": 1531332181431385, "job": 1165, "event": "compaction_finished", "compaction_time_micros": 216282, "output_level": 1, "num_output_files": 1, "total_output_size": 15292611, "num_input_records": 3584, "num_output_records": 3074, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:03:01.432011 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181432007, "job": 1165, "event": "table_file_deletion", "file_number": 10874}
>2018-07-11 18:03:01.434840 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332181434837, "job": 1165, "event": "table_file_deletion", "file_number": 10872}
>2018-07-11 18:03:06.338829 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:03:10.932650 7f29149e4700  1 mon.magna102@0(leader).log v21837 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21837)
>2018-07-11 18:03:28.250850 7f29149e4700  1 mon.magna102@0(leader).log v21838 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21838)
>2018-07-11 18:04:06.338972 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:05:06.339149 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:05:30.086964 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10876. Immutable memtables: 0.
>
>2018-07-11 18:05:30.087008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1166] Syncing log #10873
>2018-07-11 18:05:30.103505 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.086999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:05:30.103516 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1166] Flushing memtable with next log file: 10876
>
>2018-07-11 18:05:30.103530 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330103522, "job": 1166, "event": "flush_started", "num_memtables": 1, "num_entries": 789, "num_deletes": 250, "memory_usage": 1158944}
>2018-07-11 18:05:30.103532 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1166] Level-0 flush table #10877: started
>2018-07-11 18:05:30.145686 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330145672, "cf_name": "default", "job": 1166, "event": "table_file_creation", "file_number": 10877, "file_size": 642712, "table_properties": {"data_size": 636403, "index_size": 2740, "filter_size": 2650, "raw_key_size": 7463, "raw_average_key_size": 21, "raw_value_size": 629716, "raw_average_value_size": 1825, "num_data_blocks": 86, "num_entries": 345, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:05:30.145696 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1166] Level-0 flush table #10877: 642712 bytes OK
>2018-07-11 18:05:30.195765 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.162277) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10877 started
>2018-07-11 18:05:30.195773 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.195729) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10877: memtable #1 done
>2018-07-11 18:05:30.195775 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.195743) EVENT_LOG_v1 {"time_micros": 1531332330195737, "job": 1166, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:05:30.195777 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.195755) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:05:30.195783 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1166] Try to delete WAL files size 1141914, prev total WAL file size 1141914, number of live WAL files 2.
>
>2018-07-11 18:05:30.196085 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:05:30.196135 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.196129) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:05:30.196144 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1167] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:05:30.196150 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1166 Base level 0, inputs: [10877(627KB)], [10875(14MB)]
>
>2018-07-11 18:05:30.196163 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330196153, "job": 1167, "event": "compaction_started", "files_L0": [10877], "files_L1": [10875], "score": -1, "input_data_size": 15935323}
>2018-07-11 18:05:30.388143 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1167] Generated table #10878: 2979 keys, 14682393 bytes
>2018-07-11 18:05:30.388206 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330388163, "cf_name": "default", "job": 1167, "event": "table_file_creation", "file_number": 10878, "file_size": 14682393, "table_properties": {"data_size": 14569940, "index_size": 64284, "filter_size": 47242, "raw_key_size": 61025, "raw_average_key_size": 20, "raw_value_size": 14482183, "raw_average_value_size": 4861, "num_data_blocks": 1872, "num_entries": 2979, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:05:30.421331 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1167] Compacted 1@0 + 1@1 files to L1 => 14682393 bytes
>2018-07-11 18:05:30.463144 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.463072) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 70.8 rd, 65.2 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.6) out(14.0), read-write-amplify(47.6) write-amplify(22.8) OK, records in: 3419, records dropped: 440
>
>2018-07-11 18:05:30.463154 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:05:30.463094) EVENT_LOG_v1 {"time_micros": 1531332330463084, "job": 1167, "event": "compaction_finished", "compaction_time_micros": 225152, "output_level": 1, "num_output_files": 1, "total_output_size": 14682393, "num_input_records": 3419, "num_output_records": 2979, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:05:30.463350 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330463346, "job": 1167, "event": "table_file_deletion", "file_number": 10877}
>2018-07-11 18:05:30.466135 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332330466132, "job": 1167, "event": "table_file_deletion", "file_number": 10875}
>2018-07-11 18:05:30.466202 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:06:06.339331 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:07:06.339510 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:08:06.339683 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:09:06.339880 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:10:06.340060 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:11:06.340239 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:11:15.260918 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10879. Immutable memtables: 0.
>
>2018-07-11 18:11:15.260963 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1168] Syncing log #10876
>2018-07-11 18:11:15.277512 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.260953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:11:15.277524 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1168] Flushing memtable with next log file: 10879
>
>2018-07-11 18:11:15.277550 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675277542, "job": 1168, "event": "flush_started", "num_memtables": 1, "num_entries": 1467, "num_deletes": 251, "memory_usage": 2597088}
>2018-07-11 18:11:15.277554 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1168] Level-0 flush table #10880: started
>2018-07-11 18:11:15.353055 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675353040, "cf_name": "default", "job": 1168, "event": "table_file_creation", "file_number": 10880, "file_size": 2476582, "table_properties": {"data_size": 2455622, "index_size": 11476, "filter_size": 8560, "raw_key_size": 12462, "raw_average_key_size": 20, "raw_value_size": 2439007, "raw_average_value_size": 4038, "num_data_blocks": 348, "num_entries": 604, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:11:15.353066 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1168] Level-0 flush table #10880: 2476582 bytes OK
>2018-07-11 18:11:15.403088 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.369496) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10880 started
>2018-07-11 18:11:15.403096 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.403051) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10880: memtable #1 done
>2018-07-11 18:11:15.403098 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.403066) EVENT_LOG_v1 {"time_micros": 1531332675403059, "job": 1168, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:11:15.403100 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.403078) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:11:15.403119 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1168] Try to delete WAL files size 2567358, prev total WAL file size 2567358, number of live WAL files 2.
>
>2018-07-11 18:11:15.403775 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:11:15.403837 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.403830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:11:15.403844 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1169] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:11:15.403850 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1168 Base level 0, inputs: [10880(2418KB)], [10878(14MB)]
>
>2018-07-11 18:11:15.403863 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675403853, "job": 1169, "event": "compaction_started", "files_L0": [10880], "files_L1": [10878], "score": -1, "input_data_size": 17158975}
>2018-07-11 18:11:15.587130 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1169] Generated table #10881: 3075 keys, 15208518 bytes
>2018-07-11 18:11:15.587174 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675587147, "cf_name": "default", "job": 1169, "event": "table_file_creation", "file_number": 10881, "file_size": 15208518, "table_properties": {"data_size": 15090682, "index_size": 67787, "filter_size": 49122, "raw_key_size": 63291, "raw_average_key_size": 20, "raw_value_size": 14999027, "raw_average_value_size": 4877, "num_data_blocks": 1968, "num_entries": 3075, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:11:15.620284 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1169] Compacted 1@0 + 1@1 files to L1 => 15208518 bytes
>2018-07-11 18:11:15.653808 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.653742) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.3 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(2.4, 14.0) out(14.5), read-write-amplify(13.1) write-amplify(6.1) OK, records in: 3583, records dropped: 508
>
>2018-07-11 18:11:15.653817 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:11:15.653763) EVENT_LOG_v1 {"time_micros": 1531332675653753, "job": 1169, "event": "compaction_finished", "compaction_time_micros": 216403, "output_level": 1, "num_output_files": 1, "total_output_size": 15208518, "num_input_records": 3583, "num_output_records": 3075, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:11:15.654381 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675654377, "job": 1169, "event": "table_file_deletion", "file_number": 10880}
>2018-07-11 18:11:15.657286 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332675657270, "job": 1169, "event": "table_file_deletion", "file_number": 10878}
>2018-07-11 18:12:06.340422 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:12:34.294383 7f29149e4700  1 mon.magna102@0(leader).log v21839 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21839)
>2018-07-11 18:13:06.340604 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:13:50.317860 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10882. Immutable memtables: 0.
>
>2018-07-11 18:13:50.317905 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1170] Syncing log #10879
>2018-07-11 18:13:50.334380 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.317894) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:13:50.334392 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1170] Flushing memtable with next log file: 10882
>
>2018-07-11 18:13:50.334418 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830334410, "job": 1170, "event": "flush_started", "num_memtables": 1, "num_entries": 802, "num_deletes": 250, "memory_usage": 1185176}
>2018-07-11 18:13:50.334421 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1170] Level-0 flush table #10883: started
>2018-07-11 18:13:50.384911 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830384897, "cf_name": "default", "job": 1170, "event": "table_file_creation", "file_number": 10883, "file_size": 647146, "table_properties": {"data_size": 640793, "index_size": 2769, "filter_size": 2665, "raw_key_size": 7462, "raw_average_key_size": 21, "raw_value_size": 634087, "raw_average_value_size": 1837, "num_data_blocks": 87, "num_entries": 345, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:13:50.384921 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1170] Level-0 flush table #10883: 647146 bytes OK
>2018-07-11 18:13:50.443261 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.401450) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10883 started
>2018-07-11 18:13:50.443269 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.443226) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10883: memtable #1 done
>2018-07-11 18:13:50.443272 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.443240) EVENT_LOG_v1 {"time_micros": 1531332830443234, "job": 1170, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:13:50.443286 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.443252) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:13:50.443293 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1170] Try to delete WAL files size 1168106, prev total WAL file size 1168106, number of live WAL files 2.
>
>2018-07-11 18:13:50.443715 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:13:50.443912 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 18:13:50.443919 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 94857.3 total, 649.3 interval
>Cumulative writes: 121K writes, 568K keys, 121K commit groups, 1.0 writes per commit group, ingest: 0.93 GB, 0.01 MB/s
>Cumulative WAL: 121K writes, 121K syncs, 1.00 writes per sync, written: 0.93 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 660 writes, 3058 keys, 660 commit groups, 1.0 writes per commit group, ingest: 4.65 MB, 0.01 MB/s
>Interval WAL: 660 writes, 660 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   631.98 KB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       586    0.079       0      0
>  L1      1/1   14.50 MB   0.0      9.2     0.8      8.4       8.4     -0.0       0.0  10.7     72.6     66.4       130       583    0.223   2094K   254K
> Sum      2/2   15.12 MB   0.0      9.2     0.8      8.4       9.2      0.8       0.0  11.7     53.6     53.6       176      1169    0.151   2094K   254K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  13.0     54.8     53.3         1         6    0.146     10K   1458
>Uptime(secs): 94857.3 total, 94857.3 interval
>Flush(GB): cumulative 0.788, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.23 GB write, 0.10 MB/s write, 9.23 GB read, 0.10 MB/s read, 176.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   631.98 KB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       586    0.079       0      0
>  L1      1/1   14.50 MB   0.0      9.2     0.8      8.4       8.4     -0.0       0.0  10.7     72.6     66.4       130       583    0.223   2094K   254K
> Sum      2/2   15.12 MB   0.0      9.2     0.8      8.4       9.2      0.8       0.0  11.7     53.6     53.6       176      1169    0.151   2094K   254K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 94857.3 total, 94857.3 interval
>Flush(GB): cumulative 0.788, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.23 GB write, 0.10 MB/s write, 9.23 GB read, 0.10 MB/s read, 176.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 18:13:50.443940 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.443933) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:13:50.443945 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1171] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:13:50.443950 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1170 Base level 0, inputs: [10883(631KB)], [10881(14MB)]
>
>2018-07-11 18:13:50.443963 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830443954, "job": 1171, "event": "compaction_started", "files_L0": [10883], "files_L1": [10881], "score": -1, "input_data_size": 15855664}
>2018-07-11 18:13:50.635755 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1171] Generated table #10884: 2983 keys, 14636839 bytes
>2018-07-11 18:13:50.635798 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830635772, "cf_name": "default", "job": 1171, "event": "table_file_creation", "file_number": 10884, "file_size": 14636839, "table_properties": {"data_size": 14524347, "index_size": 64385, "filter_size": 47180, "raw_key_size": 61106, "raw_average_key_size": 20, "raw_value_size": 14436449, "raw_average_value_size": 4839, "num_data_blocks": 1875, "num_entries": 2983, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:13:50.678502 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1171] Compacted 1@0 + 1@1 files to L1 => 14636839 bytes
>2018-07-11 18:13:50.712019 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.711951) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 67.6 rd, 62.4 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.5) out(14.0), read-write-amplify(47.1) write-amplify(22.6) OK, records in: 3420, records dropped: 437
>
>2018-07-11 18:13:50.712028 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:13:50.711973) EVENT_LOG_v1 {"time_micros": 1531332830711963, "job": 1171, "event": "compaction_finished", "compaction_time_micros": 234522, "output_level": 1, "num_output_files": 1, "total_output_size": 14636839, "num_input_records": 3420, "num_output_records": 2983, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:13:50.712239 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830712234, "job": 1171, "event": "table_file_deletion", "file_number": 10883}
>2018-07-11 18:13:50.715056 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531332830715053, "job": 1171, "event": "table_file_deletion", "file_number": 10881}
>2018-07-11 18:13:50.715169 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:14:06.340780 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:15:06.340958 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:16:06.341146 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:17:06.341294 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:18:06.341481 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:19:06.341658 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:19:31.483095 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10885. Immutable memtables: 0.
>
>2018-07-11 18:19:31.483138 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1172] Syncing log #10882
>2018-07-11 18:19:31.499658 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.483128) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:19:31.499669 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1172] Flushing memtable with next log file: 10885
>
>2018-07-11 18:19:31.499695 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171499687, "job": 1172, "event": "flush_started", "num_memtables": 1, "num_entries": 1453, "num_deletes": 251, "memory_usage": 2567680}
>2018-07-11 18:19:31.499699 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1172] Level-0 flush table #10886: started
>2018-07-11 18:19:31.558548 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171558533, "cf_name": "default", "job": 1172, "event": "table_file_creation", "file_number": 10886, "file_size": 2448113, "table_properties": {"data_size": 2427375, "index_size": 11342, "filter_size": 8472, "raw_key_size": 12378, "raw_average_key_size": 20, "raw_value_size": 2410917, "raw_average_value_size": 4018, "num_data_blocks": 344, "num_entries": 600, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:19:31.558559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1172] Level-0 flush table #10886: 2448113 bytes OK
>2018-07-11 18:19:31.608558 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.575063) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10886 started
>2018-07-11 18:19:31.608566 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.608522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10886: memtable #1 done
>2018-07-11 18:19:31.608569 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.608536) EVENT_LOG_v1 {"time_micros": 1531333171608530, "job": 1172, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:19:31.608583 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.608548) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:19:31.608589 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1172] Try to delete WAL files size 2537820, prev total WAL file size 2537820, number of live WAL files 2.
>
>2018-07-11 18:19:31.609237 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:19:31.609289 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.609281) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:19:31.609298 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1173] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:19:31.609304 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1172 Base level 0, inputs: [10886(2390KB)], [10884(13MB)]
>
>2018-07-11 18:19:31.609318 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171609308, "job": 1173, "event": "compaction_started", "files_L0": [10886], "files_L1": [10884], "score": -1, "input_data_size": 17084952}
>2018-07-11 18:19:31.800923 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1173] Generated table #10887: 3075 keys, 15129682 bytes
>2018-07-11 18:19:31.800972 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171800944, "cf_name": "default", "job": 1173, "event": "table_file_creation", "file_number": 10887, "file_size": 15129682, "table_properties": {"data_size": 15012042, "index_size": 67753, "filter_size": 48960, "raw_key_size": 63288, "raw_average_key_size": 20, "raw_value_size": 14920398, "raw_average_value_size": 4852, "num_data_blocks": 1967, "num_entries": 3075, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:19:31.834116 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1173] Compacted 1@0 + 1@1 files to L1 => 15129682 bytes
>2018-07-11 18:19:31.867656 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.867588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.0 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(2.3, 14.0) out(14.4), read-write-amplify(13.2) write-amplify(6.2) OK, records in: 3583, records dropped: 508
>
>2018-07-11 18:19:31.867665 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:19:31.867609) EVENT_LOG_v1 {"time_micros": 1531333171867600, "job": 1173, "event": "compaction_finished", "compaction_time_micros": 224784, "output_level": 1, "num_output_files": 1, "total_output_size": 15129682, "num_input_records": 3583, "num_output_records": 3075, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:19:31.868224 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171868220, "job": 1173, "event": "table_file_deletion", "file_number": 10886}
>2018-07-11 18:19:31.871087 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333171871084, "job": 1173, "event": "table_file_deletion", "file_number": 10884}
>2018-07-11 18:20:06.341818 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:21:06.342003 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:22:06.342190 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:22:10.516811 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10888. Immutable memtables: 0.
>
>2018-07-11 18:22:10.516855 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1174] Syncing log #10885
>2018-07-11 18:22:10.533343 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.516846) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:22:10.533354 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1174] Flushing memtable with next log file: 10888
>
>2018-07-11 18:22:10.533380 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330533372, "job": 1174, "event": "flush_started", "num_memtables": 1, "num_entries": 815, "num_deletes": 250, "memory_usage": 1199272}
>2018-07-11 18:22:10.533383 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1174] Level-0 flush table #10889: started
>2018-07-11 18:22:10.575534 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330575521, "cf_name": "default", "job": 1174, "event": "table_file_creation", "file_number": 10889, "file_size": 646423, "table_properties": {"data_size": 640030, "index_size": 2803, "filter_size": 2671, "raw_key_size": 7480, "raw_average_key_size": 21, "raw_value_size": 633285, "raw_average_value_size": 1830, "num_data_blocks": 88, "num_entries": 346, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:22:10.575544 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1174] Level-0 flush table #10889: 646423 bytes OK
>2018-07-11 18:22:10.625586 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.592074) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10889 started
>2018-07-11 18:22:10.625593 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.625550) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10889: memtable #1 done
>2018-07-11 18:22:10.625596 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.625565) EVENT_LOG_v1 {"time_micros": 1531333330625558, "job": 1174, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:22:10.625598 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.625576) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:22:10.625616 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1174] Try to delete WAL files size 1182034, prev total WAL file size 1182034, number of live WAL files 2.
>
>2018-07-11 18:22:10.626037 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:22:10.626100 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.626093) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:22:10.626106 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1175] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:22:10.626113 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1174 Base level 0, inputs: [10889(631KB)], [10887(14MB)]
>
>2018-07-11 18:22:10.626126 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330626116, "job": 1175, "event": "compaction_started", "files_L0": [10889], "files_L1": [10887], "score": -1, "input_data_size": 15776105}
>2018-07-11 18:22:10.809616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1175] Generated table #10890: 2986 keys, 14570755 bytes
>2018-07-11 18:22:10.809660 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330809633, "cf_name": "default", "job": 1175, "event": "table_file_creation", "file_number": 10890, "file_size": 14570755, "table_properties": {"data_size": 14458345, "index_size": 64420, "filter_size": 47063, "raw_key_size": 61162, "raw_average_key_size": 20, "raw_value_size": 14370382, "raw_average_value_size": 4812, "num_data_blocks": 1876, "num_entries": 2986, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:22:10.842809 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1175] Compacted 1@0 + 1@1 files to L1 => 14570755 bytes
>2018-07-11 18:22:10.876318 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.876248) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 72.8 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.4) out(13.9), read-write-amplify(46.9) write-amplify(22.5) OK, records in: 3421, records dropped: 435
>
>2018-07-11 18:22:10.876327 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:22:10.876270) EVENT_LOG_v1 {"time_micros": 1531333330876260, "job": 1175, "event": "compaction_finished", "compaction_time_micros": 216664, "output_level": 1, "num_output_files": 1, "total_output_size": 14570755, "num_input_records": 3421, "num_output_records": 2986, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:22:10.876535 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330876531, "job": 1175, "event": "table_file_deletion", "file_number": 10889}
>2018-07-11 18:22:10.879356 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333330879353, "job": 1175, "event": "table_file_deletion", "file_number": 10887}
>2018-07-11 18:22:10.879473 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:23:06.342350 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:24:06.342506 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:25:06.342663 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:26:06.342848 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:27:06.343035 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8794 MB, avail 861 GB
>2018-07-11 18:27:47.696825 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10891. Immutable memtables: 0.
>
>2018-07-11 18:27:47.696870 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1176] Syncing log #10888
>2018-07-11 18:27:47.713328 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.696861) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:27:47.713341 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1176] Flushing memtable with next log file: 10891
>
>2018-07-11 18:27:47.713360 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333667713350, "job": 1176, "event": "flush_started", "num_memtables": 1, "num_entries": 1439, "num_deletes": 251, "memory_usage": 2537752}
>2018-07-11 18:27:47.713363 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1176] Level-0 flush table #10892: started
>2018-07-11 18:27:47.772247 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333667772232, "cf_name": "default", "job": 1176, "event": "table_file_creation", "file_number": 10892, "file_size": 2419647, "table_properties": {"data_size": 2399138, "index_size": 11205, "filter_size": 8380, "raw_key_size": 12294, "raw_average_key_size": 20, "raw_value_size": 2382827, "raw_average_value_size": 3998, "num_data_blocks": 340, "num_entries": 596, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:27:47.772257 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1176] Level-0 flush table #10892: 2419647 bytes OK
>2018-07-11 18:27:47.831830 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.788778) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10892 started
>2018-07-11 18:27:47.831838 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.831793) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10892: memtable #1 done
>2018-07-11 18:27:47.831841 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.831808) EVENT_LOG_v1 {"time_micros": 1531333667831802, "job": 1176, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:27:47.831856 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.831820) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:27:47.831863 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1176] Try to delete WAL files size 2508282, prev total WAL file size 2508282, number of live WAL files 2.
>
>2018-07-11 18:27:47.832609 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:27:47.832804 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 18:27:47.832811 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 95694.7 total, 837.4 interval
>Cumulative writes: 121K writes, 571K keys, 121K commit groups, 1.0 writes per commit group, ingest: 0.94 GB, 0.01 MB/s
>Cumulative WAL: 121K writes, 121K syncs, 1.00 writes per sync, written: 0.94 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 846 writes, 3707 keys, 846 commit groups, 1.0 writes per commit group, ingest: 5.94 MB, 0.01 MB/s
>Interval WAL: 846 writes, 846 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.31 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       589    0.079       0      0
>  L1      1/1   13.90 MB   0.0      9.3     0.8      8.5       8.5     -0.0       0.0  10.7     72.6     66.4       131       586    0.223   2105K   255K
> Sum      2/2   16.20 MB   0.0      9.3     0.8      8.5       9.3      0.8       0.0  11.7     53.6     53.6       177      1175    0.151   2105K   255K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.0     52.5     53.7         1         6    0.148     10K   1380
>Uptime(secs): 95694.7 total, 95694.7 interval
>Flush(GB): cumulative 0.794, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.28 GB write, 0.10 MB/s write, 9.28 GB read, 0.10 MB/s read, 177.3 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.31 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        46       589    0.079       0      0
>  L1      1/1   13.90 MB   0.0      9.3     0.8      8.5       8.5     -0.0       0.0  10.7     72.6     66.4       131       586    0.223   2105K   255K
> Sum      2/2   16.20 MB   0.0      9.3     0.8      8.5       9.3      0.8       0.0  11.7     53.6     53.6       177      1175    0.151   2105K   255K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 95694.7 total, 95694.7 interval
>Flush(GB): cumulative 0.794, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.28 GB write, 0.10 MB/s write, 9.28 GB read, 0.10 MB/s read, 177.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 18:27:47.832831 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:27:47.832825) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:27:47.832836 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1177] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:27:47.832842 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1176 Base level 0, inputs: [10892(2362KB)], [10890(13MB)]
>
>2018-07-11 18:27:47.832854 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333667832845, "job": 1177, "event": "compaction_started", "files_L0": [10892], "files_L1": [10890], "score": -1, "input_data_size": 16990402}
>2018-07-11 18:27:48.015885 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1177] Generated table #10893: 3074 keys, 15107370 bytes
>2018-07-11 18:27:48.015929 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333668015902, "cf_name": "default", "job": 1177, "event": "table_file_creation", "file_number": 10893, "file_size": 15107370, "table_properties": {"data_size": 14989895, "index_size": 67653, "filter_size": 48895, "raw_key_size": 63260, "raw_average_key_size": 20, "raw_value_size": 14898338, "raw_average_value_size": 4846, "num_data_blocks": 1964, "num_entries": 3074, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:27:48.057413 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1177] Compacted 1@0 + 1@1 files to L1 => 15107370 bytes
>2018-07-11 18:27:48.090941 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:27:48.090874) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.7 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(2.3, 13.9) out(14.4), read-write-amplify(13.3) write-amplify(6.2) OK, records in: 3582, records dropped: 508
>
>2018-07-11 18:27:48.090949 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:27:48.090895) EVENT_LOG_v1 {"time_micros": 1531333668090885, "job": 1177, "event": "compaction_finished", "compaction_time_micros": 224543, "output_level": 1, "num_output_files": 1, "total_output_size": 15107370, "num_input_records": 3582, "num_output_records": 3074, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:27:48.091551 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333668091546, "job": 1177, "event": "table_file_deletion", "file_number": 10892}
>2018-07-11 18:27:48.094434 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333668094418, "job": 1177, "event": "table_file_deletion", "file_number": 10890}
>2018-07-11 18:28:06.343204 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:29:06.343372 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:30:06.343540 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:30:30.730791 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10894. Immutable memtables: 0.
>
>2018-07-11 18:30:30.730836 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1178] Syncing log #10891
>2018-07-11 18:30:30.747286 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.730825) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:30:30.747297 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1178] Flushing memtable with next log file: 10894
>
>2018-07-11 18:30:30.747324 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333830747315, "job": 1178, "event": "flush_started", "num_memtables": 1, "num_entries": 822, "num_deletes": 250, "memory_usage": 1227016}
>2018-07-11 18:30:30.747327 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1178] Level-0 flush table #10895: started
>2018-07-11 18:30:30.789504 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333830789490, "cf_name": "default", "job": 1178, "event": "table_file_creation", "file_number": 10895, "file_size": 659474, "table_properties": {"data_size": 652977, "index_size": 2867, "filter_size": 2711, "raw_key_size": 7430, "raw_average_key_size": 21, "raw_value_size": 646239, "raw_average_value_size": 1878, "num_data_blocks": 90, "num_entries": 344, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:30:30.789514 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1178] Level-0 flush table #10895: 659474 bytes OK
>2018-07-11 18:30:30.847878 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.806043) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10895 started
>2018-07-11 18:30:30.847885 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.847842) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10895: memtable #1 done
>2018-07-11 18:30:30.847888 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.847856) EVENT_LOG_v1 {"time_micros": 1531333830847850, "job": 1178, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:30:30.847904 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.847868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:30:30.847910 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1178] Try to delete WAL files size 1209620, prev total WAL file size 1209620, number of live WAL files 2.
>
>2018-07-11 18:30:30.848288 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:30:30.848341 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:30:30.848331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:30:30.848350 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1179] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:30:30.848357 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1178 Base level 0, inputs: [10895(644KB)], [10893(14MB)]
>
>2018-07-11 18:30:30.848370 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333830848360, "job": 1179, "event": "compaction_started", "files_L0": [10895], "files_L1": [10893], "score": -1, "input_data_size": 15766844}
>2018-07-11 18:30:31.040282 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1179] Generated table #10896: 2987 keys, 14575117 bytes
>2018-07-11 18:30:31.040319 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333831040303, "cf_name": "default", "job": 1179, "event": "table_file_creation", "file_number": 10896, "file_size": 14575117, "table_properties": {"data_size": 14462656, "index_size": 64454, "filter_size": 47080, "raw_key_size": 61182, "raw_average_key_size": 20, "raw_value_size": 14374656, "raw_average_value_size": 4812, "num_data_blocks": 1877, "num_entries": 2987, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:30:31.081724 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1179] Compacted 1@0 + 1@1 files to L1 => 14575117 bytes
>2018-07-11 18:30:31.123590 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:30:31.123518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 67.6 rd, 62.5 wr, level 1, files in(1, 1) out(1) MB in(0.6, 14.4) out(13.9), read-write-amplify(46.0) write-amplify(22.1) OK, records in: 3418, records dropped: 431
>
>2018-07-11 18:30:31.123598 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:30:31.123542) EVENT_LOG_v1 {"time_micros": 1531333831123530, "job": 1179, "event": "compaction_finished", "compaction_time_micros": 233338, "output_level": 1, "num_output_files": 1, "total_output_size": 14575117, "num_input_records": 3418, "num_output_records": 2987, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:30:31.123807 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333831123802, "job": 1179, "event": "table_file_deletion", "file_number": 10895}
>2018-07-11 18:30:31.126611 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531333831126608, "job": 1179, "event": "table_file_deletion", "file_number": 10893}
>2018-07-11 18:30:31.126724 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:31:06.343706 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:32:06.343863 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:33:06.344031 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:34:06.344223 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:34:45.663305 7f29149e4700  1 mon.magna102@0(leader).log v21840 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21840)
>2018-07-11 18:35:06.344367 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:35:40.900611 7f29149e4700  1 mon.magna102@0(leader).log v21841 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21841)
>2018-07-11 18:36:01.903780 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10897. Immutable memtables: 0.
>
>2018-07-11 18:36:01.903825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1180] Syncing log #10894
>2018-07-11 18:36:01.920330 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:36:01.903815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:36:01.920342 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1180] Flushing memtable with next log file: 10897
>
>2018-07-11 18:36:01.920355 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334161920348, "job": 1180, "event": "flush_started", "num_memtables": 1, "num_entries": 1434, "num_deletes": 251, "memory_usage": 2527472}
>2018-07-11 18:36:01.920358 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1180] Level-0 flush table #10898: started
>2018-07-11 18:36:01.979254 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334161979239, "cf_name": "default", "job": 1180, "event": "table_file_creation", "file_number": 10898, "file_size": 2411904, "table_properties": {"data_size": 2391494, "index_size": 11138, "filter_size": 8348, "raw_key_size": 12341, "raw_average_key_size": 20, "raw_value_size": 2375180, "raw_average_value_size": 3971, "num_data_blocks": 338, "num_entries": 598, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:36:01.979264 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1180] Level-0 flush table #10898: 2411904 bytes OK
>2018-07-11 18:36:02.029291 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:36:01.995774) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10898 started
>2018-07-11 18:36:02.029299 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.029254) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10898: memtable #1 done
>2018-07-11 18:36:02.029302 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.029269) EVENT_LOG_v1 {"time_micros": 1531334162029262, "job": 1180, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:36:02.029305 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.029281) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:36:02.029311 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1180] Try to delete WAL files size 2498875, prev total WAL file size 2498875, number of live WAL files 2.
>
>2018-07-11 18:36:02.029953 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:36:02.030008 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.030001) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:36:02.030017 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1181] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:36:02.030024 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1180 Base level 0, inputs: [10898(2355KB)], [10896(13MB)]
>
>2018-07-11 18:36:02.030036 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334162030027, "job": 1181, "event": "compaction_started", "files_L0": [10898], "files_L1": [10896], "score": -1, "input_data_size": 16987021}
>2018-07-11 18:36:02.213342 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1181] Generated table #10899: 3075 keys, 15105608 bytes
>2018-07-11 18:36:02.213372 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334162213357, "cf_name": "default", "job": 1181, "event": "table_file_creation", "file_number": 10899, "file_size": 15105608, "table_properties": {"data_size": 14988174, "index_size": 67620, "filter_size": 48887, "raw_key_size": 63276, "raw_average_key_size": 20, "raw_value_size": 14896632, "raw_average_value_size": 4844, "num_data_blocks": 1963, "num_entries": 3075, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:36:02.246512 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1181] Compacted 1@0 + 1@1 files to L1 => 15105608 bytes
>2018-07-11 18:36:02.280016 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.279949) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.5 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(2.3, 13.9) out(14.4), read-write-amplify(13.3) write-amplify(6.3) OK, records in: 3585, records dropped: 510
>
>2018-07-11 18:36:02.280025 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:36:02.279971) EVENT_LOG_v1 {"time_micros": 1531334162279961, "job": 1181, "event": "compaction_finished", "compaction_time_micros": 216462, "output_level": 1, "num_output_files": 1, "total_output_size": 15105608, "num_input_records": 3585, "num_output_records": 3075, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:36:02.280581 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334162280577, "job": 1181, "event": "table_file_deletion", "file_number": 10898}
>2018-07-11 18:36:02.283461 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334162283458, "job": 1181, "event": "table_file_deletion", "file_number": 10896}
>2018-07-11 18:36:06.344533 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:37:06.344711 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:37:33.263094 7f29149e4700  1 mon.magna102@0(leader).log v21842 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21842)
>2018-07-11 18:38:06.344870 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:38:50.961910 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10900. Immutable memtables: 0.
>
>2018-07-11 18:38:50.961954 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1182] Syncing log #10897
>2018-07-11 18:38:50.978451 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:38:50.961945) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:38:50.978463 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1182] Flushing memtable with next log file: 10900
>
>2018-07-11 18:38:50.978477 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334330978469, "job": 1182, "event": "flush_started", "num_memtables": 1, "num_entries": 851, "num_deletes": 250, "memory_usage": 1289504}
>2018-07-11 18:38:50.978479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1182] Level-0 flush table #10901: started
>2018-07-11 18:38:51.029020 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334331029005, "cf_name": "default", "job": 1182, "event": "table_file_creation", "file_number": 10901, "file_size": 699286, "table_properties": {"data_size": 692544, "index_size": 2991, "filter_size": 2832, "raw_key_size": 7602, "raw_average_key_size": 21, "raw_value_size": 685576, "raw_average_value_size": 1947, "num_data_blocks": 94, "num_entries": 352, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:38:51.029030 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1182] Level-0 flush table #10901: 699286 bytes OK
>2018-07-11 18:38:51.079059 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.045577) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10901 started
>2018-07-11 18:38:51.079068 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.079023) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10901: memtable #1 done
>2018-07-11 18:38:51.079070 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.079038) EVENT_LOG_v1 {"time_micros": 1531334331079031, "job": 1182, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:38:51.079072 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.079049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:38:51.079078 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1182] Try to delete WAL files size 1271373, prev total WAL file size 1271373, number of live WAL files 2.
>
>2018-07-11 18:38:51.079430 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:38:51.079607 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 18:38:51.079614 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 96357.9 total, 663.2 interval
>Cumulative writes: 122K writes, 575K keys, 122K commit groups, 1.0 writes per commit group, ingest: 0.94 GB, 0.01 MB/s
>Cumulative WAL: 122K writes, 122K syncs, 1.00 writes per sync, written: 0.94 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 674 writes, 3107 keys, 674 commit groups, 1.0 writes per commit group, ingest: 4.75 MB, 0.01 MB/s
>Interval WAL: 674 writes, 674 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   682.90 KB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        47       592    0.079       0      0
>  L1      1/1   14.41 MB   0.0      9.3     0.8      8.5       8.5     -0.0       0.0  10.7     72.6     66.4       132       589    0.223   2115K   257K
> Sum      2/2   15.07 MB   0.0      9.3     0.8      8.5       9.3      0.8       0.0  11.7     53.6     53.6       178      1181    0.151   2115K   257K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.9     54.2     52.9         1         6    0.146     10K   1449
>Uptime(secs): 96357.9 total, 96357.9 interval
>Flush(GB): cumulative 0.797, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.32 GB write, 0.10 MB/s write, 9.33 GB read, 0.10 MB/s read, 178.1 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   682.90 KB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        47       592    0.079       0      0
>  L1      1/1   14.41 MB   0.0      9.3     0.8      8.5       8.5     -0.0       0.0  10.7     72.6     66.4       132       589    0.223   2115K   257K
> Sum      2/2   15.07 MB   0.0      9.3     0.8      8.5       9.3      0.8       0.0  11.7     53.6     53.6       178      1181    0.151   2115K   257K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 96357.9 total, 96357.9 interval
>Flush(GB): cumulative 0.797, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.32 GB write, 0.10 MB/s write, 9.33 GB read, 0.10 MB/s read, 178.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 18:38:51.079634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.079628) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:38:51.079639 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1183] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:38:51.079645 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1182 Base level 0, inputs: [10901(682KB)], [10899(14MB)]
>
>2018-07-11 18:38:51.079658 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334331079649, "job": 1183, "event": "compaction_started", "files_L0": [10901], "files_L1": [10899], "score": -1, "input_data_size": 15804894}
>2018-07-11 18:38:51.254773 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1183] Generated table #10902: 2997 keys, 14633508 bytes
>2018-07-11 18:38:51.254821 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334331254793, "cf_name": "default", "job": 1183, "event": "table_file_creation", "file_number": 10902, "file_size": 14633508, "table_properties": {"data_size": 14520624, "index_size": 64691, "filter_size": 47266, "raw_key_size": 61385, "raw_average_key_size": 20, "raw_value_size": 14432317, "raw_average_value_size": 4815, "num_data_blocks": 1884, "num_entries": 2997, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:38:51.296302 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1183] Compacted 1@0 + 1@1 files to L1 => 14633508 bytes
>2018-07-11 18:38:51.329751 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.329681) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.0 rd, 67.6 wr, level 1, files in(1, 1) out(1) MB in(0.7, 14.4) out(14.0), read-write-amplify(43.5) write-amplify(20.9) OK, records in: 3427, records dropped: 430
>
>2018-07-11 18:38:51.329761 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:38:51.329703) EVENT_LOG_v1 {"time_micros": 1531334331329693, "job": 1183, "event": "compaction_finished", "compaction_time_micros": 216627, "output_level": 1, "num_output_files": 1, "total_output_size": 14633508, "num_input_records": 3427, "num_output_records": 2997, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:38:51.329955 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334331329950, "job": 1183, "event": "table_file_deletion", "file_number": 10901}
>2018-07-11 18:38:51.332835 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334331332820, "job": 1183, "event": "table_file_deletion", "file_number": 10899}
>2018-07-11 18:38:51.332867 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:39:06.345052 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:40:06.345230 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:41:06.345411 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:42:06.345596 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:43:06.345776 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:43:47.115634 7f29149e4700  1 mon.magna102@0(leader).log v21843 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21843)
>2018-07-11 18:43:57.149813 7f29149e4700  1 mon.magna102@0(leader).log v21844 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21844)
>2018-07-11 18:44:06.345968 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:44:14.143160 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10903. Immutable memtables: 0.
>
>2018-07-11 18:44:14.143238 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1184] Syncing log #10900
>2018-07-11 18:44:14.159736 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.143229) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:44:14.159747 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1184] Flushing memtable with next log file: 10903
>
>2018-07-11 18:44:14.159773 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654159765, "job": 1184, "event": "flush_started", "num_memtables": 1, "num_entries": 1406, "num_deletes": 251, "memory_usage": 2468336}
>2018-07-11 18:44:14.159777 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1184] Level-0 flush table #10904: started
>2018-07-11 18:44:14.218611 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654218596, "cf_name": "default", "job": 1184, "event": "table_file_creation", "file_number": 10904, "file_size": 2354565, "table_properties": {"data_size": 2334609, "index_size": 10868, "filter_size": 8164, "raw_key_size": 12173, "raw_average_key_size": 20, "raw_value_size": 2318596, "raw_average_value_size": 3929, "num_data_blocks": 330, "num_entries": 590, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:44:14.218622 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1184] Level-0 flush table #10904: 2354565 bytes OK
>2018-07-11 18:44:14.268617 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.235137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10904 started
>2018-07-11 18:44:14.268626 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.268581) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10904: memtable #1 done
>2018-07-11 18:44:14.268628 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.268595) EVENT_LOG_v1 {"time_micros": 1531334654268589, "job": 1184, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:44:14.268643 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.268607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:44:14.268649 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1184] Try to delete WAL files size 2439393, prev total WAL file size 2439393, number of live WAL files 2.
>
>2018-07-11 18:44:14.269312 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:44:14.269363 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.269356) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:44:14.269373 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1185] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:44:14.269379 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1184 Base level 0, inputs: [10904(2299KB)], [10902(13MB)]
>
>2018-07-11 18:44:14.269393 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654269383, "job": 1185, "event": "compaction_started", "files_L0": [10904], "files_L1": [10902], "score": -1, "input_data_size": 16988073}
>2018-07-11 18:44:14.460937 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1185] Generated table #10905: 3077 keys, 15114613 bytes
>2018-07-11 18:44:14.460983 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654460967, "cf_name": "default", "job": 1185, "event": "table_file_creation", "file_number": 10905, "file_size": 15114613, "table_properties": {"data_size": 14997117, "index_size": 67652, "filter_size": 48917, "raw_key_size": 63311, "raw_average_key_size": 20, "raw_value_size": 14905523, "raw_average_value_size": 4844, "num_data_blocks": 1964, "num_entries": 3077, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:44:14.494115 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1185] Compacted 1@0 + 1@1 files to L1 => 15114613 bytes
>2018-07-11 18:44:14.527638 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.527566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.6 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.0) out(14.4), read-write-amplify(13.6) write-amplify(6.4) OK, records in: 3587, records dropped: 510
>
>2018-07-11 18:44:14.527647 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:44:14.527588) EVENT_LOG_v1 {"time_micros": 1531334654527579, "job": 1185, "event": "compaction_finished", "compaction_time_micros": 224706, "output_level": 1, "num_output_files": 1, "total_output_size": 15114613, "num_input_records": 3587, "num_output_records": 3077, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:44:14.528167 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654528150, "job": 1185, "event": "table_file_deletion", "file_number": 10904}
>2018-07-11 18:44:14.531154 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334654531151, "job": 1185, "event": "table_file_deletion", "file_number": 10902}
>2018-07-11 18:44:57.182413 7f29149e4700  1 mon.magna102@0(leader).log v21845 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21845)
>2018-07-11 18:45:06.346139 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:45:10.018836 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 3 slow requests are blocked > 32 sec. Implicated osds 3,5 (REQUEST_SLOW)
>2018-07-11 18:45:10.191992 7f29149e4700  1 mon.magna102@0(leader).log v21846 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21846)
>2018-07-11 18:45:16.021642 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 6 slow requests are blocked > 32 sec. Implicated osds 3,5,6 (REQUEST_SLOW)
>2018-07-11 18:45:16.176023 7f29149e4700  1 mon.magna102@0(leader).log v21847 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21847)
>2018-07-11 18:45:21.066382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 8 slow requests are blocked > 32 sec. Implicated osds 3,5,6 (REQUEST_SLOW)
>2018-07-11 18:45:21.159677 7f29149e4700  1 mon.magna102@0(leader).log v21848 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21848)
>2018-07-11 18:45:30.027728 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 14 slow requests are blocked > 32 sec. Implicated osds 3,5,6 (REQUEST_SLOW)
>2018-07-11 18:45:30.177281 7f29149e4700  1 mon.magna102@0(leader).log v21849 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21849)
>2018-07-11 18:45:36.030572 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 16 slow requests are blocked > 32 sec. Implicated osds 3,5,6 (REQUEST_SLOW)
>2018-07-11 18:45:36.177960 7f29149e4700  1 mon.magna102@0(leader).log v21850 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21850)
>2018-07-11 18:45:41.067157 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 20 slow requests are blocked > 32 sec. Implicated osds 3,5,6 (REQUEST_SLOW)
>2018-07-11 18:45:41.161739 7f29149e4700  1 mon.magna102@0(leader).log v21851 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21851)
>2018-07-11 18:46:06.346327 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:46:26.053010 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:46:26.208155 7f29149e4700  1 mon.magna102@0(leader).log v21852 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21852)
>2018-07-11 18:46:30.054588 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:46:30.208579 7f29149e4700  1 mon.magna102@0(leader).log v21853 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21853)
>2018-07-11 18:46:46.061818 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:46:46.218204 7f29149e4700  1 mon.magna102@0(leader).log v21854 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21854)
>2018-07-11 18:46:51.101377 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:46:51.219889 7f29149e4700  1 mon.magna102@0(leader).log v21855 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21855)
>2018-07-11 18:47:06.346487 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:47:11.171733 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10906. Immutable memtables: 0.
>
>2018-07-11 18:47:11.171776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1186] Syncing log #10903
>2018-07-11 18:47:11.188264 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.171766) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:47:11.188275 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1186] Flushing memtable with next log file: 10906
>
>2018-07-11 18:47:11.188302 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831188294, "job": 1186, "event": "flush_started", "num_memtables": 1, "num_entries": 959, "num_deletes": 250, "memory_usage": 1557512}
>2018-07-11 18:47:11.188305 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1186] Level-0 flush table #10907: started
>2018-07-11 18:47:11.230454 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831230440, "cf_name": "default", "job": 1186, "event": "table_file_creation", "file_number": 10907, "file_size": 921784, "table_properties": {"data_size": 913572, "index_size": 3788, "filter_size": 3505, "raw_key_size": 8292, "raw_average_key_size": 21, "raw_value_size": 905559, "raw_average_value_size": 2346, "num_data_blocks": 118, "num_entries": 386, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:47:11.230464 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1186] Level-0 flush table #10907: 921784 bytes OK
>2018-07-11 18:47:11.288831 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.246995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10907 started
>2018-07-11 18:47:11.288839 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.288795) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10907: memtable #1 done
>2018-07-11 18:47:11.288841 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.288809) EVENT_LOG_v1 {"time_micros": 1531334831288803, "job": 1186, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:47:11.288843 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.288821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:47:11.288862 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1186] Try to delete WAL files size 1537864, prev total WAL file size 1537864, number of live WAL files 2.
>
>2018-07-11 18:47:11.289299 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:47:11.289349 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.289342) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:47:11.289358 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1187] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:47:11.289365 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1186 Base level 0, inputs: [10907(900KB)], [10905(14MB)]
>
>2018-07-11 18:47:11.289377 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831289368, "job": 1187, "event": "compaction_started", "files_L0": [10907], "files_L1": [10905], "score": -1, "input_data_size": 16036397}
>2018-07-11 18:47:11.472855 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1187] Generated table #10908: 3037 keys, 14892111 bytes
>2018-07-11 18:47:11.472899 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831472871, "cf_name": "default", "job": 1187, "event": "table_file_creation", "file_number": 10908, "file_size": 14892111, "table_properties": {"data_size": 14777449, "index_size": 65683, "filter_size": 48052, "raw_key_size": 62198, "raw_average_key_size": 20, "raw_value_size": 14687883, "raw_average_value_size": 4836, "num_data_blocks": 1913, "num_entries": 3037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:47:11.514387 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1187] Compacted 1@0 + 1@1 files to L1 => 14892111 bytes
>2018-07-11 18:47:11.547912 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.547845) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.3 rd, 66.2 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.4) out(14.2), read-write-amplify(33.6) write-amplify(16.2) OK, records in: 3463, records dropped: 426
>
>2018-07-11 18:47:11.547920 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:47:11.547866) EVENT_LOG_v1 {"time_micros": 1531334831547856, "job": 1187, "event": "compaction_finished", "compaction_time_micros": 224993, "output_level": 1, "num_output_files": 1, "total_output_size": 14892111, "num_input_records": 3463, "num_output_records": 3037, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:47:11.548207 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831548202, "job": 1187, "event": "table_file_deletion", "file_number": 10907}
>2018-07-11 18:47:11.550976 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531334831550973, "job": 1187, "event": "table_file_deletion", "file_number": 10905}
>2018-07-11 18:47:11.551092 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:48:06.346666 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:48:12.100545 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:48:12.254115 7f29149e4700  1 mon.magna102@0(leader).log v21856 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21856)
>2018-07-11 18:48:16.102478 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:48:16.254569 7f29149e4700  1 mon.magna102@0(leader).log v21857 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21857)
>2018-07-11 18:48:26.107004 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:48:26.272092 7f29149e4700  1 mon.magna102@0(leader).log v21858 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21858)
>2018-07-11 18:48:31.147011 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:48:31.264280 7f29149e4700  1 mon.magna102@0(leader).log v21859 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21859)
>2018-07-11 18:49:06.346837 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:49:50.144708 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:49:50.288107 7f29149e4700  1 mon.magna102@0(leader).log v21860 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21860)
>2018-07-11 18:49:54.146382 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:49:54.296848 7f29149e4700  1 mon.magna102@0(leader).log v21861 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21861)
>2018-07-11 18:50:06.347019 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:50:10.153707 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:50:10.308006 7f29149e4700  1 mon.magna102@0(leader).log v21862 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21862)
>2018-07-11 18:51:06.347207 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:51:30.189507 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:51:30.339999 7f29149e4700  1 mon.magna102@0(leader).log v21863 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21863)
>2018-07-11 18:51:34.191146 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:51:34.340482 7f29149e4700  1 mon.magna102@0(leader).log v21864 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21864)
>2018-07-11 18:51:50.198337 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:51:50.358527 7f29149e4700  1 mon.magna102@0(leader).log v21865 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21865)
>2018-07-11 18:51:50.425564 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10909. Immutable memtables: 0.
>
>2018-07-11 18:51:50.425607 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1188] Syncing log #10906
>2018-07-11 18:51:50.442158 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.425598) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:51:50.442169 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1188] Flushing memtable with next log file: 10909
>
>2018-07-11 18:51:50.442203 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110442195, "job": 1188, "event": "flush_started", "num_memtables": 1, "num_entries": 1316, "num_deletes": 251, "memory_usage": 2345632}
>2018-07-11 18:51:50.442206 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1188] Level-0 flush table #10910: started
>2018-07-11 18:51:50.492701 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110492686, "cf_name": "default", "job": 1188, "event": "table_file_creation", "file_number": 10910, "file_size": 2226301, "table_properties": {"data_size": 2207756, "index_size": 9937, "filter_size": 7684, "raw_key_size": 11737, "raw_average_key_size": 20, "raw_value_size": 2192697, "raw_average_value_size": 3846, "num_data_blocks": 302, "num_entries": 570, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:51:50.492711 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1188] Level-0 flush table #10910: 2226301 bytes OK
>2018-07-11 18:51:50.551066 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.509223) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10910 started
>2018-07-11 18:51:50.551074 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.551029) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10910: memtable #1 done
>2018-07-11 18:51:50.551077 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.551044) EVENT_LOG_v1 {"time_micros": 1531335110551038, "job": 1188, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:51:50.551091 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.551056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:51:50.551098 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1188] Try to delete WAL files size 2319653, prev total WAL file size 2319653, number of live WAL files 2.
>
>2018-07-11 18:51:50.551732 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:51:50.551926 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 18:51:50.551933 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 97137.4 total, 779.5 interval
>Cumulative writes: 123K writes, 578K keys, 123K commit groups, 1.0 writes per commit group, ingest: 0.95 GB, 0.01 MB/s
>Cumulative WAL: 123K writes, 123K syncs, 1.00 writes per sync, written: 0.95 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 832 writes, 3681 keys, 832 commit groups, 1.0 writes per commit group, ingest: 6.01 MB, 0.01 MB/s
>Interval WAL: 832 writes, 832 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.12 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        47       595    0.079       0      0
>  L1      1/1   14.20 MB   0.0      9.4     0.8      8.6       8.6     -0.0       0.0  10.7     72.6     66.4       132       592    0.223   2126K   258K
> Sum      2/2   16.33 MB   0.0      9.4     0.8      8.6       9.4      0.8       0.0  11.7     53.6     53.6       179      1187    0.151   2126K   258K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.1     53.7     55.1         1         6    0.145     10K   1366
>Uptime(secs): 97137.4 total, 97137.4 interval
>Flush(GB): cumulative 0.802, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.37 GB write, 0.10 MB/s write, 9.37 GB read, 0.10 MB/s read, 179.0 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.12 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        47       595    0.079       0      0
>  L1      1/1   14.20 MB   0.0      9.4     0.8      8.6       8.6     -0.0       0.0  10.7     72.6     66.4       132       592    0.223   2126K   258K
> Sum      2/2   16.33 MB   0.0      9.4     0.8      8.6       9.4      0.8       0.0  11.7     53.6     53.6       179      1187    0.151   2126K   258K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 97137.4 total, 97137.4 interval
>Flush(GB): cumulative 0.802, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.37 GB write, 0.10 MB/s write, 9.37 GB read, 0.10 MB/s read, 179.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 18:51:50.551954 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.551947) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:51:50.551960 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1189] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:51:50.551966 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1188 Base level 0, inputs: [10910(2174KB)], [10908(14MB)]
>
>2018-07-11 18:51:50.551979 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110551969, "job": 1189, "event": "compaction_started", "files_L0": [10910], "files_L1": [10908], "score": -1, "input_data_size": 17118412}
>2018-07-11 18:51:50.735031 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1189] Generated table #10911: 3097 keys, 15235578 bytes
>2018-07-11 18:51:50.735077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110735061, "cf_name": "default", "job": 1189, "event": "table_file_creation", "file_number": 10911, "file_size": 15235578, "table_properties": {"data_size": 15117791, "index_size": 67668, "filter_size": 49192, "raw_key_size": 63688, "raw_average_key_size": 20, "raw_value_size": 15025913, "raw_average_value_size": 4851, "num_data_blocks": 1964, "num_entries": 3097, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:51:50.768215 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1189] Compacted 1@0 + 1@1 files to L1 => 15235578 bytes
>2018-07-11 18:51:50.810076 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.810004) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 70.5 wr, level 1, files in(1, 1) out(1) MB in(2.1, 14.2) out(14.5), read-write-amplify(14.5) write-amplify(6.8) OK, records in: 3607, records dropped: 510
>
>2018-07-11 18:51:50.810086 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:51:50.810026) EVENT_LOG_v1 {"time_micros": 1531335110810016, "job": 1189, "event": "compaction_finished", "compaction_time_micros": 216219, "output_level": 1, "num_output_files": 1, "total_output_size": 15235578, "num_input_records": 3607, "num_output_records": 3097, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:51:50.810590 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110810574, "job": 1189, "event": "table_file_deletion", "file_number": 10910}
>2018-07-11 18:51:50.813618 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335110813615, "job": 1189, "event": "table_file_deletion", "file_number": 10908}
>2018-07-11 18:52:06.347393 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:53:06.347579 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:53:14.236113 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:53:14.385668 7f29149e4700  1 mon.magna102@0(leader).log v21866 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21866)
>2018-07-11 18:53:16.237163 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 18:53:16.385986 7f29149e4700  1 mon.magna102@0(leader).log v21867 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21867)
>2018-07-11 18:53:30.243324 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:53:30.395572 7f29149e4700  1 mon.magna102@0(leader).log v21868 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21868)
>2018-07-11 18:54:06.347757 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:54:54.280902 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:54:54.452709 7f29149e4700  1 mon.magna102@0(leader).log v21869 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21869)
>2018-07-11 18:54:56.281987 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 18:54:56.453038 7f29149e4700  1 mon.magna102@0(leader).log v21870 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21870)
>2018-07-11 18:55:06.347931 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:55:10.288103 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:55:10.437578 7f29149e4700  1 mon.magna102@0(leader).log v21871 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21871)
>2018-07-11 18:55:16.329006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:55:16.446536 7f29149e4700  1 mon.magna102@0(leader).log v21872 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21872)
>2018-07-11 18:55:31.399251 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10912. Immutable memtables: 0.
>
>2018-07-11 18:55:31.399297 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1190] Syncing log #10909
>2018-07-11 18:55:31.415720 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.399288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:55:31.415732 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1190] Flushing memtable with next log file: 10912
>
>2018-07-11 18:55:31.415745 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331415738, "job": 1190, "event": "flush_started", "num_memtables": 1, "num_entries": 1081, "num_deletes": 250, "memory_usage": 1833232}
>2018-07-11 18:55:31.415748 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1190] Level-0 flush table #10913: started
>2018-07-11 18:55:31.466270 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331466255, "cf_name": "default", "job": 1190, "event": "table_file_creation", "file_number": 10913, "file_size": 1025340, "table_properties": {"data_size": 1016377, "index_size": 4222, "filter_size": 3821, "raw_key_size": 8488, "raw_average_key_size": 21, "raw_value_size": 1007907, "raw_average_value_size": 2545, "num_data_blocks": 132, "num_entries": 396, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 18:55:31.466280 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1190] Level-0 flush table #10913: 1025340 bytes OK
>2018-07-11 18:55:31.524613 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.482787) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10913 started
>2018-07-11 18:55:31.524621 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.524578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10913: memtable #1 done
>2018-07-11 18:55:31.524624 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.524592) EVENT_LOG_v1 {"time_micros": 1531335331524586, "job": 1190, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:55:31.524626 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.524604) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:55:31.524632 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1190] Try to delete WAL files size 1811747, prev total WAL file size 1811747, number of live WAL files 2.
>
>2018-07-11 18:55:31.525118 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:55:31.525171 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.525164) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 18:55:31.525190 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1191] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:55:31.525209 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1190 Base level 0, inputs: [10913(1001KB)], [10911(14MB)]
>
>2018-07-11 18:55:31.525223 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331525213, "job": 1191, "event": "compaction_started", "files_L0": [10913], "files_L1": [10911], "score": -1, "input_data_size": 16260918}
>2018-07-11 18:55:31.716862 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1191] Generated table #10914: 3089 keys, 15254589 bytes
>2018-07-11 18:55:31.716912 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331716884, "cf_name": "default", "job": 1191, "event": "table_file_creation", "file_number": 10914, "file_size": 15254589, "table_properties": {"data_size": 15137632, "index_size": 66916, "filter_size": 49114, "raw_key_size": 63255, "raw_average_key_size": 20, "raw_value_size": 15046476, "raw_average_value_size": 4870, "num_data_blocks": 1949, "num_entries": 3089, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:55:31.758504 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1191] Compacted 1@0 + 1@1 files to L1 => 15254589 bytes
>2018-07-11 18:55:31.800391 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.800322) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.7 rd, 65.4 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.5) out(14.5), read-write-amplify(30.7) write-amplify(14.9) OK, records in: 3493, records dropped: 404
>
>2018-07-11 18:55:31.800399 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:55:31.800343) EVENT_LOG_v1 {"time_micros": 1531335331800334, "job": 1191, "event": "compaction_finished", "compaction_time_micros": 233265, "output_level": 1, "num_output_files": 1, "total_output_size": 15254589, "num_input_records": 3493, "num_output_records": 3089, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:55:31.800651 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331800646, "job": 1191, "event": "table_file_deletion", "file_number": 10913}
>2018-07-11 18:55:31.803444 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335331803441, "job": 1191, "event": "table_file_deletion", "file_number": 10911}
>2018-07-11 18:55:31.803495 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:56:06.348118 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:56:30.324302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:56:30.481444 7f29149e4700  1 mon.magna102@0(leader).log v21873 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21873)
>2018-07-11 18:56:40.328834 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:56:40.473895 7f29149e4700  1 mon.magna102@0(leader).log v21874 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21874)
>2018-07-11 18:56:56.335997 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:56:56.491869 7f29149e4700  1 mon.magna102@0(leader).log v21875 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21875)
>2018-07-11 18:57:01.375032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:57:01.475666 7f29149e4700  1 mon.magna102@0(leader).log v21876 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21876)
>2018-07-11 18:57:06.384289 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:58:06.398050 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:58:16.371809 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:58:16.532403 7f29149e4700  1 mon.magna102@0(leader).log v21877 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21877)
>2018-07-11 18:58:20.373547 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 18:58:20.524626 7f29149e4700  1 mon.magna102@0(leader).log v21878 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21878)
>2018-07-11 18:58:36.381071 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 18:58:36.527533 7f29149e4700  1 mon.magna102@0(leader).log v21879 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21879)
>2018-07-11 18:58:41.416359 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 18:58:41.544699 7f29149e4700  1 mon.magna102@0(leader).log v21880 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21880)
>2018-07-11 18:59:06.438275 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 18:59:38.541233 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10915. Immutable memtables: 0.
>
>2018-07-11 18:59:38.541276 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1192] Syncing log #10912
>2018-07-11 18:59:38.557635 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.541267) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 18:59:38.557647 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1192] Flushing memtable with next log file: 10915
>
>2018-07-11 18:59:38.557661 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578557653, "job": 1192, "event": "flush_started", "num_memtables": 1, "num_entries": 1188, "num_deletes": 251, "memory_usage": 2064880}
>2018-07-11 18:59:38.557664 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1192] Level-0 flush table #10916: started
>2018-07-11 18:59:38.609422 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578609408, "cf_name": "default", "job": 1192, "event": "table_file_creation", "file_number": 10916, "file_size": 1958561, "table_properties": {"data_size": 1942065, "index_size": 8737, "filter_size": 6839, "raw_key_size": 10943, "raw_average_key_size": 20, "raw_value_size": 1928397, "raw_average_value_size": 3624, "num_data_blocks": 266, "num_entries": 532, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 18:59:38.609433 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1192] Level-0 flush table #10916: 1958561 bytes OK
>2018-07-11 18:59:38.659479 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.625990) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10916 started
>2018-07-11 18:59:38.659488 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.659442) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10916: memtable #1 done
>2018-07-11 18:59:38.659490 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.659457) EVENT_LOG_v1 {"time_micros": 1531335578659451, "job": 1192, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 18:59:38.659492 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.659469) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 18:59:38.659501 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1192] Try to delete WAL files size 2041221, prev total WAL file size 2041221, number of live WAL files 2.
>
>2018-07-11 18:59:38.660040 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 18:59:38.660095 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.660088) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 18:59:38.660103 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1193] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 18:59:38.660109 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1192 Base level 0, inputs: [10916(1912KB)], [10914(14MB)]
>
>2018-07-11 18:59:38.660122 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578660112, "job": 1193, "event": "compaction_started", "files_L0": [10916], "files_L1": [10914], "score": -1, "input_data_size": 17213150}
>2018-07-11 18:59:38.843536 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1193] Generated table #10917: 3111 keys, 15329167 bytes
>2018-07-11 18:59:38.843585 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578843558, "cf_name": "default", "job": 1193, "event": "table_file_creation", "file_number": 10917, "file_size": 15329167, "table_properties": {"data_size": 15211147, "index_size": 67682, "filter_size": 49411, "raw_key_size": 63951, "raw_average_key_size": 20, "raw_value_size": 15119060, "raw_average_value_size": 4859, "num_data_blocks": 1964, "num_entries": 3111, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 18:59:38.876731 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1193] Compacted 1@0 + 1@1 files to L1 => 15329167 bytes
>2018-07-11 18:59:38.910246 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.910172) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 70.8 wr, level 1, files in(1, 1) out(1) MB in(1.9, 14.5) out(14.6), read-write-amplify(16.6) write-amplify(7.8) OK, records in: 3621, records dropped: 510
>
>2018-07-11 18:59:38.910255 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-18:59:38.910199) EVENT_LOG_v1 {"time_micros": 1531335578910190, "job": 1193, "event": "compaction_finished", "compaction_time_micros": 216594, "output_level": 1, "num_output_files": 1, "total_output_size": 15329167, "num_input_records": 3621, "num_output_records": 3111, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 18:59:38.910685 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578910681, "job": 1193, "event": "table_file_deletion", "file_number": 10916}
>2018-07-11 18:59:38.913686 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335578913681, "job": 1193, "event": "table_file_deletion", "file_number": 10914}
>2018-07-11 18:59:56.417221 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 18:59:56.570464 7f29149e4700  1 mon.magna102@0(leader).log v21881 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21881)
>2018-07-11 19:00:00.418947 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 19:00:00.579289 7f29149e4700  1 mon.magna102@0(leader).log v21882 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21882)
>2018-07-11 19:00:06.462933 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:00:14.425270 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 19:00:14.597299 7f29149e4700  1 mon.magna102@0(leader).log v21883 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21883)
>2018-07-11 19:00:21.464141 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 19:00:21.589513 7f29149e4700  1 mon.magna102@0(leader).log v21884 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21884)
>2018-07-11 19:01:06.485862 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:01:36.462452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 19:01:36.621722 7f29149e4700  1 mon.magna102@0(leader).log v21885 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21885)
>2018-07-11 19:01:40.464315 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 19:01:40.615030 7f29149e4700  1 mon.magna102@0(leader).log v21886 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21886)
>2018-07-11 19:01:47.623897 7f29149e4700  1 mon.magna102@0(leader).log v21887 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21887)
>2018-07-11 19:02:06.508478 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:02:23.618196 7f29149e4700  1 mon.magna102@0(leader).log v21888 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21888)
>2018-07-11 19:03:06.538942 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:03:51.634516 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10918. Immutable memtables: 0.
>
>2018-07-11 19:03:51.634593 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1194] Syncing log #10915
>2018-07-11 19:03:51.650770 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.634583) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:03:51.650782 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1194] Flushing memtable with next log file: 10918
>
>2018-07-11 19:03:51.650795 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335831650788, "job": 1194, "event": "flush_started", "num_memtables": 1, "num_entries": 1201, "num_deletes": 250, "memory_usage": 2071480}
>2018-07-11 19:03:51.650798 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1194] Level-0 flush table #10919: started
>2018-07-11 19:03:51.701330 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335831701316, "cf_name": "default", "job": 1194, "event": "table_file_creation", "file_number": 10919, "file_size": 1154029, "table_properties": {"data_size": 1144076, "index_size": 4800, "filter_size": 4233, "raw_key_size": 8869, "raw_average_key_size": 21, "raw_value_size": 1134928, "raw_average_value_size": 2734, "num_data_blocks": 150, "num_entries": 415, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:03:51.701340 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1194] Level-0 flush table #10919: 1154029 bytes OK
>2018-07-11 19:03:51.751381 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.717887) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10919 started
>2018-07-11 19:03:51.751389 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.751345) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10919: memtable #1 done
>2018-07-11 19:03:51.751392 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.751359) EVENT_LOG_v1 {"time_micros": 1531335831751353, "job": 1194, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:03:51.751394 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.751371) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:03:51.751400 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1194] Try to delete WAL files size 2047204, prev total WAL file size 2047204, number of live WAL files 2.
>
>2018-07-11 19:03:51.751938 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:03:51.752119 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 19:03:51.752126 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 97858.6 total, 721.2 interval
>Cumulative writes: 124K writes, 582K keys, 124K commit groups, 1.0 writes per commit group, ingest: 0.95 GB, 0.01 MB/s
>Cumulative WAL: 124K writes, 124K syncs, 1.00 writes per sync, written: 0.95 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 772 writes, 3470 keys, 772 commit groups, 1.0 writes per commit group, ingest: 5.63 MB, 0.01 MB/s
>Interval WAL: 772 writes, 772 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.10 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       598    0.079       0      0
>  L1      1/1   14.62 MB   0.0      9.4     0.8      8.6       8.6     -0.0       0.0  10.7     72.6     66.4       133       595    0.223   2136K   259K
> Sum      2/2   15.72 MB   0.0      9.4     0.8      8.6       9.4      0.8       0.0  11.7     53.6     53.6       180      1193    0.151   2136K   259K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.1     55.5     54.8         1         6    0.145     10K   1424
>Uptime(secs): 97858.6 total, 97858.6 interval
>Flush(GB): cumulative 0.806, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.42 GB write, 0.10 MB/s write, 9.42 GB read, 0.10 MB/s read, 179.9 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.10 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       598    0.079       0      0
>  L1      1/1   14.62 MB   0.0      9.4     0.8      8.6       8.6     -0.0       0.0  10.7     72.6     66.4       133       595    0.223   2136K   259K
> Sum      2/2   15.72 MB   0.0      9.4     0.8      8.6       9.4      0.8       0.0  11.7     53.6     53.6       180      1193    0.151   2136K   259K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 97858.6 total, 97858.6 interval
>Flush(GB): cumulative 0.806, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.42 GB write, 0.10 MB/s write, 9.42 GB read, 0.10 MB/s read, 179.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 19:03:51.752146 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:03:51.752139) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:03:51.752150 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1195] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:03:51.752156 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1194 Base level 0, inputs: [10919(1126KB)], [10917(14MB)]
>
>2018-07-11 19:03:51.752168 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335831752159, "job": 1195, "event": "compaction_started", "files_L0": [10919], "files_L1": [10917], "score": -1, "input_data_size": 16483196}
>2018-07-11 19:03:51.935427 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1195] Generated table #10920: 3138 keys, 15586508 bytes
>2018-07-11 19:03:51.935464 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335831935449, "cf_name": "default", "job": 1195, "event": "table_file_creation", "file_number": 10920, "file_size": 15586508, "table_properties": {"data_size": 15467404, "index_size": 68080, "filter_size": 50097, "raw_key_size": 64251, "raw_average_key_size": 20, "raw_value_size": 15374730, "raw_average_value_size": 4899, "num_data_blocks": 1983, "num_entries": 3138, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:03:51.969822 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1195] Compacted 1@0 + 1@1 files to L1 => 15586508 bytes
>2018-07-11 19:03:52.011704 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:03:52.011633) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.7 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.6) out(14.9), read-write-amplify(27.8) write-amplify(13.5) OK, records in: 3526, records dropped: 388
>
>2018-07-11 19:03:52.011713 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:03:52.011655) EVENT_LOG_v1 {"time_micros": 1531335832011645, "job": 1195, "event": "compaction_finished", "compaction_time_micros": 217637, "output_level": 1, "num_output_files": 1, "total_output_size": 15586508, "num_input_records": 3526, "num_output_records": 3138, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:03:52.011987 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335832011983, "job": 1195, "event": "table_file_deletion", "file_number": 10919}
>2018-07-11 19:03:52.014802 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531335832014799, "job": 1195, "event": "table_file_deletion", "file_number": 10917}
>2018-07-11 19:03:52.014856 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:04:06.563129 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:05:06.593504 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:06:06.614307 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:07:06.644606 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:07:40.773684 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10921. Immutable memtables: 0.
>
>2018-07-11 19:07:40.773728 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1196] Syncing log #10918
>2018-07-11 19:07:40.790263 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.773718) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:07:40.790274 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1196] Flushing memtable with next log file: 10921
>
>2018-07-11 19:07:40.790301 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336060790293, "job": 1196, "event": "flush_started", "num_memtables": 1, "num_entries": 1061, "num_deletes": 251, "memory_usage": 1733080}
>2018-07-11 19:07:40.790304 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1196] Level-0 flush table #10922: started
>2018-07-11 19:07:40.840805 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336060840790, "cf_name": "default", "job": 1196, "event": "table_file_creation", "file_number": 10922, "file_size": 1651053, "table_properties": {"data_size": 1636599, "index_size": 7614, "filter_size": 5920, "raw_key_size": 10026, "raw_average_key_size": 20, "raw_value_size": 1624397, "raw_average_value_size": 3328, "num_data_blocks": 232, "num_entries": 488, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:07:40.840816 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1196] Level-0 flush table #10922: 1651053 bytes OK
>2018-07-11 19:07:40.890824 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.857313) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10922 started
>2018-07-11 19:07:40.890833 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.890788) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10922: memtable #1 done
>2018-07-11 19:07:40.890835 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.890803) EVENT_LOG_v1 {"time_micros": 1531336060890796, "job": 1196, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:07:40.890837 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.890815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:07:40.890855 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1196] Try to delete WAL files size 1710756, prev total WAL file size 1710756, number of live WAL files 2.
>
>2018-07-11 19:07:40.891322 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:07:40.891373 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:07:40.891366) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:07:40.891383 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1197] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:07:40.891389 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1196 Base level 0, inputs: [10922(1612KB)], [10920(14MB)]
>
>2018-07-11 19:07:40.891403 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336060891393, "job": 1197, "event": "compaction_started", "files_L0": [10922], "files_L1": [10920], "score": -1, "input_data_size": 17237561}
>2018-07-11 19:07:41.083225 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1197] Generated table #10923: 3118 keys, 15280018 bytes
>2018-07-11 19:07:41.083275 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336061083258, "cf_name": "default", "job": 1197, "event": "table_file_creation", "file_number": 10923, "file_size": 15280018, "table_properties": {"data_size": 15162117, "index_size": 67653, "filter_size": 49321, "raw_key_size": 64081, "raw_average_key_size": 20, "raw_value_size": 15069953, "raw_average_value_size": 4833, "num_data_blocks": 1963, "num_entries": 3118, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:07:41.116379 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1197] Compacted 1@0 + 1@1 files to L1 => 15280018 bytes
>2018-07-11 19:07:41.149889 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:07:41.149822) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.6 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.9) out(14.6), read-write-amplify(19.7) write-amplify(9.3) OK, records in: 3626, records dropped: 508
>
>2018-07-11 19:07:41.149898 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:07:41.149843) EVENT_LOG_v1 {"time_micros": 1531336061149834, "job": 1197, "event": "compaction_finished", "compaction_time_micros": 224960, "output_level": 1, "num_output_files": 1, "total_output_size": 15280018, "num_input_records": 3626, "num_output_records": 3118, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:07:41.150336 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336061150332, "job": 1197, "event": "table_file_deletion", "file_number": 10922}
>2018-07-11 19:07:41.153378 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336061153364, "job": 1197, "event": "table_file_deletion", "file_number": 10920}
>2018-07-11 19:08:06.668692 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:09:06.690890 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:10:06.729954 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:11:06.752036 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:12:06.782471 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:12:11.866726 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10924. Immutable memtables: 0.
>
>2018-07-11 19:12:11.866770 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1198] Syncing log #10921
>2018-07-11 19:12:11.883213 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.866760) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:12:11.883225 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1198] Flushing memtable with next log file: 10924
>
>2018-07-11 19:12:11.883251 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336331883243, "job": 1198, "event": "flush_started", "num_memtables": 1, "num_entries": 1200, "num_deletes": 250, "memory_usage": 2031616}
>2018-07-11 19:12:11.883254 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1198] Level-0 flush table #10925: started
>2018-07-11 19:12:11.933735 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336331933721, "cf_name": "default", "job": 1198, "event": "table_file_creation", "file_number": 10925, "file_size": 1062064, "table_properties": {"data_size": 1052579, "index_size": 4588, "filter_size": 3977, "raw_key_size": 8510, "raw_average_key_size": 21, "raw_value_size": 1043841, "raw_average_value_size": 2622, "num_data_blocks": 144, "num_entries": 398, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:12:11.933745 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1198] Level-0 flush table #10925: 1062064 bytes OK
>2018-07-11 19:12:11.983791 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.950283) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10925 started
>2018-07-11 19:12:11.983799 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.983755) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10925: memtable #1 done
>2018-07-11 19:12:11.983801 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.983770) EVENT_LOG_v1 {"time_micros": 1531336331983763, "job": 1198, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:12:11.983803 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.983781) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:12:11.983822 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1198] Try to delete WAL files size 2007146, prev total WAL file size 2007146, number of live WAL files 2.
>
>2018-07-11 19:12:11.984356 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:12:11.984405 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:12:11.984398) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:12:11.984414 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1199] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:12:11.984421 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1198 Base level 0, inputs: [10925(1037KB)], [10923(14MB)]
>
>2018-07-11 19:12:11.984434 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336331984424, "job": 1199, "event": "compaction_started", "files_L0": [10925], "files_L1": [10923], "score": -1, "input_data_size": 16342082}
>2018-07-11 19:12:12.176149 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1199] Generated table #10926: 3139 keys, 15516420 bytes
>2018-07-11 19:12:12.176209 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336332176170, "cf_name": "default", "job": 1199, "event": "table_file_creation", "file_number": 10926, "file_size": 15516420, "table_properties": {"data_size": 15397412, "index_size": 68111, "filter_size": 49970, "raw_key_size": 64271, "raw_average_key_size": 20, "raw_value_size": 15304701, "raw_average_value_size": 4875, "num_data_blocks": 1984, "num_entries": 3139, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:12:12.217629 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1199] Compacted 1@0 + 1@1 files to L1 => 15516420 bytes
>2018-07-11 19:12:12.251214 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:12:12.251136) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 70.1 rd, 66.5 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.6) out(14.8), read-write-amplify(30.0) write-amplify(14.6) OK, records in: 3516, records dropped: 377
>
>2018-07-11 19:12:12.251223 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:12:12.251158) EVENT_LOG_v1 {"time_micros": 1531336332251148, "job": 1199, "event": "compaction_finished", "compaction_time_micros": 233181, "output_level": 1, "num_output_files": 1, "total_output_size": 15516420, "num_input_records": 3516, "num_output_records": 3139, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:12:12.251510 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336332251506, "job": 1199, "event": "table_file_deletion", "file_number": 10925}
>2018-07-11 19:12:12.254348 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336332254345, "job": 1199, "event": "table_file_deletion", "file_number": 10923}
>2018-07-11 19:12:12.254464 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:13:06.806688 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:14:06.835820 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:15:06.857772 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:15:59.013082 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10927. Immutable memtables: 0.
>
>2018-07-11 19:15:59.013125 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1200] Syncing log #10924
>2018-07-11 19:15:59.029610 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.013116) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:15:59.029622 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1200] Flushing memtable with next log file: 10927
>
>2018-07-11 19:15:59.029649 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559029640, "job": 1200, "event": "flush_started", "num_memtables": 1, "num_entries": 1054, "num_deletes": 251, "memory_usage": 1718264}
>2018-07-11 19:15:59.029652 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1200] Level-0 flush table #10928: started
>2018-07-11 19:15:59.088505 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559088491, "cf_name": "default", "job": 1200, "event": "table_file_creation", "file_number": 10928, "file_size": 1636824, "table_properties": {"data_size": 1622483, "index_size": 7547, "filter_size": 5874, "raw_key_size": 9984, "raw_average_key_size": 20, "raw_value_size": 1610352, "raw_average_value_size": 3313, "num_data_blocks": 230, "num_entries": 486, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:15:59.088515 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1200] Level-0 flush table #10928: 1636824 bytes OK
>2018-07-11 19:15:59.146862 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.105022) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10928 started
>2018-07-11 19:15:59.146870 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.146826) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10928: memtable #1 done
>2018-07-11 19:15:59.146873 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.146840) EVENT_LOG_v1 {"time_micros": 1531336559146834, "job": 1200, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:15:59.146875 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.146852) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:15:59.146893 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1200] Try to delete WAL files size 1695987, prev total WAL file size 1695987, number of live WAL files 2.
>
>2018-07-11 19:15:59.147361 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:15:59.147544 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 19:15:59.147550 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 98586.0 total, 727.4 interval
>Cumulative writes: 124K writes, 585K keys, 124K commit groups, 1.0 writes per commit group, ingest: 0.96 GB, 0.01 MB/s
>Cumulative WAL: 124K writes, 124K syncs, 1.00 writes per sync, written: 0.96 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 734 writes, 3315 keys, 734 commit groups, 1.0 writes per commit group, ingest: 5.16 MB, 0.01 MB/s
>Interval WAL: 734 writes, 734 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.56 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       601    0.079       0      0
>  L1      1/1   14.80 MB   0.0      9.5     0.8      8.7       8.7     -0.0       0.0  10.7     72.6     66.4       134       598    0.223   2147K   261K
> Sum      2/2   16.36 MB   0.0      9.5     0.8      8.7       9.5      0.8       0.0  11.7     53.6     53.6       181      1199    0.151   2147K   261K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.7     53.9     54.7         1         6    0.148     10K   1273
>Uptime(secs): 98586.0 total, 98586.0 interval
>Flush(GB): cumulative 0.810, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.47 GB write, 0.10 MB/s write, 9.46 GB read, 0.10 MB/s read, 180.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.56 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       601    0.079       0      0
>  L1      1/1   14.80 MB   0.0      9.5     0.8      8.7       8.7     -0.0       0.0  10.7     72.6     66.4       134       598    0.223   2147K   261K
> Sum      2/2   16.36 MB   0.0      9.5     0.8      8.7       9.5      0.8       0.0  11.7     53.6     53.6       181      1199    0.151   2147K   261K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 98586.0 total, 98586.0 interval
>Flush(GB): cumulative 0.810, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.47 GB write, 0.10 MB/s write, 9.46 GB read, 0.10 MB/s read, 180.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 19:15:59.147572 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.147566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:15:59.147577 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1201] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:15:59.147583 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1200 Base level 0, inputs: [10928(1598KB)], [10926(14MB)]
>
>2018-07-11 19:15:59.147596 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559147586, "job": 1201, "event": "compaction_started", "files_L0": [10928], "files_L1": [10926], "score": -1, "input_data_size": 17153244}
>2018-07-11 19:15:59.347564 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1201] Generated table #10929: 3117 keys, 15198031 bytes
>2018-07-11 19:15:59.347608 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559347581, "cf_name": "default", "job": 1201, "event": "table_file_creation", "file_number": 10929, "file_size": 15198031, "table_properties": {"data_size": 15080336, "index_size": 67616, "filter_size": 49152, "raw_key_size": 64059, "raw_average_key_size": 20, "raw_value_size": 14988211, "raw_average_value_size": 4808, "num_data_blocks": 1962, "num_entries": 3117, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:15:59.380758 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1201] Compacted 1@0 + 1@1 files to L1 => 15198031 bytes
>2018-07-11 19:15:59.414280 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.414212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.6 rd, 65.2 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.8) out(14.5), read-write-amplify(19.8) write-amplify(9.3) OK, records in: 3625, records dropped: 508
>
>2018-07-11 19:15:59.414288 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:15:59.414233) EVENT_LOG_v1 {"time_micros": 1531336559414224, "job": 1201, "event": "compaction_finished", "compaction_time_micros": 233146, "output_level": 1, "num_output_files": 1, "total_output_size": 15198031, "num_input_records": 3625, "num_output_records": 3117, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:15:59.414693 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559414689, "job": 1201, "event": "table_file_deletion", "file_number": 10928}
>2018-07-11 19:15:59.417748 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336559417745, "job": 1201, "event": "table_file_deletion", "file_number": 10926}
>2018-07-11 19:16:06.881785 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:16:57.020246 7f29149e4700  1 mon.magna102@0(leader).log v21889 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21889)
>2018-07-11 19:17:06.912366 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:18:06.942851 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:18:07.043194 7f29149e4700  1 mon.magna102@0(leader).log v21890 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21890)
>2018-07-11 19:19:06.965120 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:20:06.995803 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:20:32.106716 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10930. Immutable memtables: 0.
>
>2018-07-11 19:20:32.106760 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1202] Syncing log #10927
>2018-07-11 19:20:32.123243 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.106750) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:20:32.123254 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1202] Flushing memtable with next log file: 10930
>
>2018-07-11 19:20:32.123281 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832123273, "job": 1202, "event": "flush_started", "num_memtables": 1, "num_entries": 1230, "num_deletes": 250, "memory_usage": 2086424}
>2018-07-11 19:20:32.123285 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1202] Level-0 flush table #10931: started
>2018-07-11 19:20:32.173766 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832173751, "cf_name": "default", "job": 1202, "event": "table_file_creation", "file_number": 10931, "file_size": 1108601, "table_properties": {"data_size": 1098842, "index_size": 4719, "filter_size": 4120, "raw_key_size": 8793, "raw_average_key_size": 21, "raw_value_size": 1089778, "raw_average_value_size": 2651, "num_data_blocks": 148, "num_entries": 411, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:20:32.173776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1202] Level-0 flush table #10931: 1108601 bytes OK
>2018-07-11 19:20:32.223776 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.190290) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10931 started
>2018-07-11 19:20:32.223784 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.223741) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10931: memtable #1 done
>2018-07-11 19:20:32.223787 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.223755) EVENT_LOG_v1 {"time_micros": 1531336832223749, "job": 1202, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:20:32.223789 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.223767) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:20:32.223808 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1202] Try to delete WAL files size 2061081, prev total WAL file size 2061081, number of live WAL files 2.
>
>2018-07-11 19:20:32.224367 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:20:32.224418 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.224411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:20:32.224428 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1203] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:20:32.224434 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1202 Base level 0, inputs: [10931(1082KB)], [10929(14MB)]
>
>2018-07-11 19:20:32.224448 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832224438, "job": 1203, "event": "compaction_started", "files_L0": [10931], "files_L1": [10929], "score": -1, "input_data_size": 16306632}
>2018-07-11 19:20:32.449507 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1203] Generated table #10932: 3148 keys, 15487679 bytes
>2018-07-11 19:20:32.449556 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832449528, "cf_name": "default", "job": 1203, "event": "table_file_creation", "file_number": 10932, "file_size": 15487679, "table_properties": {"data_size": 15368502, "index_size": 68283, "filter_size": 49967, "raw_key_size": 64449, "raw_average_key_size": 20, "raw_value_size": 15275546, "raw_average_value_size": 4852, "num_data_blocks": 1989, "num_entries": 3148, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:20:32.482691 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1203] Compacted 1@0 + 1@1 files to L1 => 15487679 bytes
>2018-07-11 19:20:32.524556 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.524487) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 63.1 rd, 60.0 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.5) out(14.8), read-write-amplify(28.7) write-amplify(14.0) OK, records in: 3528, records dropped: 380
>
>2018-07-11 19:20:32.524566 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:20:32.524508) EVENT_LOG_v1 {"time_micros": 1531336832524499, "job": 1203, "event": "compaction_finished", "compaction_time_micros": 258227, "output_level": 1, "num_output_files": 1, "total_output_size": 15487679, "num_input_records": 3528, "num_output_records": 3148, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:20:32.524849 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832524844, "job": 1203, "event": "table_file_deletion", "file_number": 10931}
>2018-07-11 19:20:32.527689 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531336832527687, "job": 1203, "event": "table_file_deletion", "file_number": 10929}
>2018-07-11 19:20:32.527802 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:21:07.011927 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:22:07.042313 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:23:07.063054 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:24:07.093342 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:24:11.194157 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10933. Immutable memtables: 0.
>
>2018-07-11 19:24:11.194243 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1204] Syncing log #10930
>2018-07-11 19:24:11.210741 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.194233) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:24:11.210753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1204] Flushing memtable with next log file: 10933
>
>2018-07-11 19:24:11.210766 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051210759, "job": 1204, "event": "flush_started", "num_memtables": 1, "num_entries": 1026, "num_deletes": 251, "memory_usage": 1658624}
>2018-07-11 19:24:11.210769 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1204] Level-0 flush table #10934: started
>2018-07-11 19:24:11.261317 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051261302, "cf_name": "default", "job": 1204, "event": "table_file_creation", "file_number": 10934, "file_size": 1579889, "table_properties": {"data_size": 1565995, "index_size": 7284, "filter_size": 5690, "raw_key_size": 9816, "raw_average_key_size": 20, "raw_value_size": 1554172, "raw_average_value_size": 3251, "num_data_blocks": 222, "num_entries": 478, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:24:11.261328 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1204] Level-0 flush table #10934: 1579889 bytes OK
>2018-07-11 19:24:11.311314 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.277811) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10934 started
>2018-07-11 19:24:11.311322 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.311278) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10934: memtable #1 done
>2018-07-11 19:24:11.311325 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.311292) EVENT_LOG_v1 {"time_micros": 1531337051311286, "job": 1204, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:24:11.311327 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.311304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:24:11.311333 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1204] Try to delete WAL files size 1636911, prev total WAL file size 1636911, number of live WAL files 2.
>
>2018-07-11 19:24:11.311791 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:24:11.311846 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.311839) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:24:11.311855 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1205] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:24:11.311861 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1204 Base level 0, inputs: [10934(1542KB)], [10932(14MB)]
>
>2018-07-11 19:24:11.311874 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051311865, "job": 1205, "event": "compaction_started", "files_L0": [10934], "files_L1": [10932], "score": -1, "input_data_size": 17067568}
>2018-07-11 19:24:11.495358 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1205] Generated table #10935: 3118 keys, 15160013 bytes
>2018-07-11 19:24:11.495392 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051495377, "cf_name": "default", "job": 1205, "event": "table_file_creation", "file_number": 10935, "file_size": 15160013, "table_properties": {"data_size": 15042512, "index_size": 67517, "filter_size": 49057, "raw_key_size": 64069, "raw_average_key_size": 20, "raw_value_size": 14950446, "raw_average_value_size": 4794, "num_data_blocks": 1959, "num_entries": 3118, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:24:11.528529 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1205] Compacted 1@0 + 1@1 files to L1 => 15160013 bytes
>2018-07-11 19:24:11.562033 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.561964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.8 rd, 70.0 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.8) out(14.5), read-write-amplify(20.4) write-amplify(9.6) OK, records in: 3626, records dropped: 508
>
>2018-07-11 19:24:11.562042 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:24:11.561988) EVENT_LOG_v1 {"time_micros": 1531337051561976, "job": 1205, "event": "compaction_finished", "compaction_time_micros": 216638, "output_level": 1, "num_output_files": 1, "total_output_size": 15160013, "num_input_records": 3626, "num_output_records": 3118, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:24:11.562438 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051562434, "job": 1205, "event": "table_file_deletion", "file_number": 10934}
>2018-07-11 19:24:11.565465 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337051565444, "job": 1205, "event": "table_file_deletion", "file_number": 10932}
>2018-07-11 19:25:07.125873 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:26:07.147927 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:27:07.169767 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:28:07.200199 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:28:52.303928 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10936. Immutable memtables: 0.
>
>2018-07-11 19:28:52.303971 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1206] Syncing log #10933
>2018-07-11 19:28:52.320343 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.303962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:28:52.320355 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1206] Flushing memtable with next log file: 10936
>
>2018-07-11 19:28:52.320381 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332320373, "job": 1206, "event": "flush_started", "num_memtables": 1, "num_entries": 1235, "num_deletes": 250, "memory_usage": 2105928}
>2018-07-11 19:28:52.320384 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1206] Level-0 flush table #10937: started
>2018-07-11 19:28:52.362570 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332362556, "cf_name": "default", "job": 1206, "event": "table_file_creation", "file_number": 10937, "file_size": 1099340, "table_properties": {"data_size": 1089578, "index_size": 4748, "filter_size": 4094, "raw_key_size": 8610, "raw_average_key_size": 21, "raw_value_size": 1080656, "raw_average_value_size": 2681, "num_data_blocks": 149, "num_entries": 403, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:28:52.362580 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1206] Level-0 flush table #10937: 1099340 bytes OK
>2018-07-11 19:28:52.420942 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.379113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10937 started
>2018-07-11 19:28:52.420950 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.420906) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10937: memtable #1 done
>2018-07-11 19:28:52.420952 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.420921) EVENT_LOG_v1 {"time_micros": 1531337332420914, "job": 1206, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:28:52.420967 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.420932) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:28:52.420974 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1206] Try to delete WAL files size 2080991, prev total WAL file size 2080991, number of live WAL files 2.
>
>2018-07-11 19:28:52.421610 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:28:52.421804 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 19:28:52.421810 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 99359.3 total, 773.3 interval
>Cumulative writes: 125K writes, 589K keys, 125K commit groups, 1.0 writes per commit group, ingest: 0.96 GB, 0.01 MB/s
>Cumulative WAL: 125K writes, 125K syncs, 1.00 writes per sync, written: 0.96 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 784 writes, 3491 keys, 784 commit groups, 1.0 writes per commit group, ingest: 5.51 MB, 0.01 MB/s
>Interval WAL: 784 writes, 784 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.05 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       604    0.079       0      0
>  L1      1/1   14.46 MB   0.0      9.5     0.8      8.7       8.7     -0.0       0.0  10.7     72.6     66.3       134       601    0.223   2158K   262K
> Sum      2/2   15.51 MB   0.0      9.5     0.8      8.7       9.5      0.8       0.0  11.7     53.6     53.6       182      1205    0.151   2158K   262K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  13.1     53.5     52.5         1         6    0.150     10K   1396
>Uptime(secs): 99359.3 total, 99359.3 interval
>Flush(GB): cumulative 0.814, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.51 GB write, 0.10 MB/s write, 9.51 GB read, 0.10 MB/s read, 181.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.05 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        47       604    0.079       0      0
>  L1      1/1   14.46 MB   0.0      9.5     0.8      8.7       8.7     -0.0       0.0  10.7     72.6     66.3       134       601    0.223   2158K   262K
> Sum      2/2   15.51 MB   0.0      9.5     0.8      8.7       9.5      0.8       0.0  11.7     53.6     53.6       182      1205    0.151   2158K   262K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 99359.3 total, 99359.3 interval
>Flush(GB): cumulative 0.814, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.51 GB write, 0.10 MB/s write, 9.51 GB read, 0.10 MB/s read, 181.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 19:28:52.421831 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.421825) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:28:52.421836 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1207] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:28:52.421841 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1206 Base level 0, inputs: [10937(1073KB)], [10935(14MB)]
>
>2018-07-11 19:28:52.421853 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332421844, "job": 1207, "event": "compaction_started", "files_L0": [10937], "files_L1": [10935], "score": -1, "input_data_size": 16259353}
>2018-07-11 19:28:52.604972 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1207] Generated table #10938: 3149 keys, 15467591 bytes
>2018-07-11 19:28:52.605022 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332604994, "cf_name": "default", "job": 1207, "event": "table_file_creation", "file_number": 10938, "file_size": 15467591, "table_properties": {"data_size": 15348412, "index_size": 68316, "filter_size": 49936, "raw_key_size": 64469, "raw_average_key_size": 20, "raw_value_size": 15255419, "raw_average_value_size": 4844, "num_data_blocks": 1990, "num_entries": 3149, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:28:52.638184 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1207] Compacted 1@0 + 1@1 files to L1 => 15467591 bytes
>2018-07-11 19:28:52.680035 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.679966) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.2 rd, 71.5 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.5) out(14.8), read-write-amplify(28.9) write-amplify(14.1) OK, records in: 3521, records dropped: 372
>
>2018-07-11 19:28:52.680043 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:28:52.679987) EVENT_LOG_v1 {"time_micros": 1531337332679978, "job": 1207, "event": "compaction_finished", "compaction_time_micros": 216308, "output_level": 1, "num_output_files": 1, "total_output_size": 15467591, "num_input_records": 3521, "num_output_records": 3149, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:28:52.680340 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332680336, "job": 1207, "event": "table_file_deletion", "file_number": 10937}
>2018-07-11 19:28:52.683108 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337332683104, "job": 1207, "event": "table_file_deletion", "file_number": 10935}
>2018-07-11 19:28:52.683227 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:29:07.224457 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:30:06.997058 7f29149e4700  1 mon.magna102@0(leader).log v21891 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21891)
>2018-07-11 19:30:07.224637 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:30:25.223587 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:30:25.373689 7f29149e4700  1 mon.magna102@0(leader).log v21892 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21892)
>2018-07-11 19:30:31.226374 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:30:31.382727 7f29149e4700  1 mon.magna102@0(leader).log v21893 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21893)
>2018-07-11 19:31:07.224811 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:31:35.254147 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:31:35.405340 7f29149e4700  1 mon.magna102@0(leader).log v21894 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21894)
>2018-07-11 19:31:45.258469 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:31:45.406212 7f29149e4700  1 mon.magna102@0(leader).log v21895 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21895)
>2018-07-11 19:32:01.265489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:32:01.415681 7f29149e4700  1 mon.magna102@0(leader).log v21896 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21896)
>2018-07-11 19:32:07.224987 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:32:07.307717 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:32:07.424784 7f29149e4700  1 mon.magna102@0(leader).log v21897 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21897)
>2018-07-11 19:32:15.418669 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10939. Immutable memtables: 0.
>
>2018-07-11 19:32:15.418713 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1208] Syncing log #10936
>2018-07-11 19:32:15.435245 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.418703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:32:15.435257 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1208] Flushing memtable with next log file: 10939
>
>2018-07-11 19:32:15.435284 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535435276, "job": 1208, "event": "flush_started", "num_memtables": 1, "num_entries": 1026, "num_deletes": 251, "memory_usage": 1696496}
>2018-07-11 19:32:15.435287 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1208] Level-0 flush table #10940: started
>2018-07-11 19:32:15.494096 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535494082, "cf_name": "default", "job": 1208, "event": "table_file_creation", "file_number": 10940, "file_size": 1614631, "table_properties": {"data_size": 1600733, "index_size": 7223, "filter_size": 5755, "raw_key_size": 9958, "raw_average_key_size": 20, "raw_value_size": 1588829, "raw_average_value_size": 3275, "num_data_blocks": 220, "num_entries": 485, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:32:15.494107 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1208] Level-0 flush table #10940: 1614631 bytes OK
>2018-07-11 19:32:15.552467 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.510648) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10940 started
>2018-07-11 19:32:15.552475 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.552430) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10940: memtable #1 done
>2018-07-11 19:32:15.552478 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.552445) EVENT_LOG_v1 {"time_micros": 1531337535552439, "job": 1208, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:32:15.552492 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.552457) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:32:15.552499 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1208] Try to delete WAL files size 1675320, prev total WAL file size 1675320, number of live WAL files 2.
>
>2018-07-11 19:32:15.552988 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:32:15.553051 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.553044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:32:15.553057 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1209] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:32:15.553064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1208 Base level 0, inputs: [10940(1576KB)], [10938(14MB)]
>
>2018-07-11 19:32:15.553077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535553067, "job": 1209, "event": "compaction_started", "files_L0": [10940], "files_L1": [10938], "score": -1, "input_data_size": 17082222}
>2018-07-11 19:32:15.736510 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1209] Generated table #10941: 3124 keys, 15202148 bytes
>2018-07-11 19:32:15.736554 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535736527, "cf_name": "default", "job": 1209, "event": "table_file_creation", "file_number": 10941, "file_size": 15202148, "table_properties": {"data_size": 15084581, "index_size": 67491, "filter_size": 49149, "raw_key_size": 64180, "raw_average_key_size": 20, "raw_value_size": 14992450, "raw_average_value_size": 4799, "num_data_blocks": 1958, "num_entries": 3124, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:32:15.769687 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1209] Compacted 1@0 + 1@1 files to L1 => 15202148 bytes
>2018-07-11 19:32:15.811558 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.811492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.9 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.8) out(14.5), read-write-amplify(20.0) write-amplify(9.4) OK, records in: 3634, records dropped: 510
>
>2018-07-11 19:32:15.811567 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:32:15.811513) EVENT_LOG_v1 {"time_micros": 1531337535811503, "job": 1209, "event": "compaction_finished", "compaction_time_micros": 216593, "output_level": 1, "num_output_files": 1, "total_output_size": 15202148, "num_input_records": 3634, "num_output_records": 3124, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:32:15.811974 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535811970, "job": 1209, "event": "table_file_deletion", "file_number": 10940}
>2018-07-11 19:32:15.814998 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337535814995, "job": 1209, "event": "table_file_deletion", "file_number": 10938}
>2018-07-11 19:33:07.225163 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:33:21.300961 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:33:21.459730 7f29149e4700  1 mon.magna102@0(leader).log v21898 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21898)
>2018-07-11 19:33:25.302646 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:33:25.451833 7f29149e4700  1 mon.magna102@0(leader).log v21899 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21899)
>2018-07-11 19:33:45.311365 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:33:45.461638 7f29149e4700  1 mon.magna102@0(leader).log v21900 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21900)
>2018-07-11 19:34:07.225358 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:34:55.342200 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:34:55.492903 7f29149e4700  1 mon.magna102@0(leader).log v21901 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21901)
>2018-07-11 19:35:01.344719 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:35:01.501859 7f29149e4700  1 mon.magna102@0(leader).log v21902 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21902)
>2018-07-11 19:35:07.225548 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:35:25.355277 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:35:25.528652 7f29149e4700  1 mon.magna102@0(leader).log v21903 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21903)
>2018-07-11 19:36:07.225724 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:36:41.388603 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 3,6)
>2018-07-11 19:36:41.544630 7f29149e4700  1 mon.magna102@0(leader).log v21904 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21904)
>2018-07-11 19:37:05.399042 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:37:05.579694 7f29149e4700  1 mon.magna102@0(leader).log v21905 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21905)
>2018-07-11 19:37:07.225899 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 19:37:12.522386 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10942. Immutable memtables: 0.
>
>2018-07-11 19:37:12.522432 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1210] Syncing log #10939
>2018-07-11 19:37:12.538521 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.522422) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:37:12.538532 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1210] Flushing memtable with next log file: 10942
>
>2018-07-11 19:37:12.538546 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832538539, "job": 1210, "event": "flush_started", "num_memtables": 1, "num_entries": 1355, "num_deletes": 250, "memory_usage": 2436704}
>2018-07-11 19:37:12.538549 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1210] Level-0 flush table #10943: started
>2018-07-11 19:37:12.589162 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832589147, "cf_name": "default", "job": 1210, "event": "table_file_creation", "file_number": 10943, "file_size": 1340048, "table_properties": {"data_size": 1328836, "index_size": 5501, "filter_size": 4791, "raw_key_size": 9309, "raw_average_key_size": 21, "raw_value_size": 1318871, "raw_average_value_size": 3018, "num_data_blocks": 172, "num_entries": 437, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:37:12.589171 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1210] Level-0 flush table #10943: 1340048 bytes OK
>2018-07-11 19:37:12.639174 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.605704) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10943 started
>2018-07-11 19:37:12.639190 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.639138) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10943: memtable #1 done
>2018-07-11 19:37:12.639192 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.639153) EVENT_LOG_v1 {"time_micros": 1531337832639147, "job": 1210, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:37:12.639194 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.639165) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:37:12.639214 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1210] Try to delete WAL files size 2410257, prev total WAL file size 2410257, number of live WAL files 2.
>
>2018-07-11 19:37:12.639848 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:37:12.639911 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.639904) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:37:12.639918 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1211] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:37:12.639924 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1210 Base level 0, inputs: [10943(1308KB)], [10941(14MB)]
>
>2018-07-11 19:37:12.639938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832639928, "job": 1211, "event": "compaction_started", "files_L0": [10943], "files_L1": [10941], "score": -1, "input_data_size": 16542196}
>2018-07-11 19:37:12.831554 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1211] Generated table #10944: 3195 keys, 15799816 bytes
>2018-07-11 19:37:12.831599 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832831571, "cf_name": "default", "job": 1211, "event": "table_file_creation", "file_number": 10944, "file_size": 15799816, "table_properties": {"data_size": 15678578, "index_size": 69414, "filter_size": 50897, "raw_key_size": 65404, "raw_average_key_size": 20, "raw_value_size": 15584168, "raw_average_value_size": 4877, "num_data_blocks": 2022, "num_entries": 3195, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:37:12.873071 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1211] Compacted 1@0 + 1@1 files to L1 => 15799816 bytes
>2018-07-11 19:37:12.914937 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.914871) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.0 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.5) out(15.1), read-write-amplify(24.1) write-amplify(11.8) OK, records in: 3561, records dropped: 366
>
>2018-07-11 19:37:12.914946 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:37:12.914892) EVENT_LOG_v1 {"time_micros": 1531337832914883, "job": 1211, "event": "compaction_finished", "compaction_time_micros": 233118, "output_level": 1, "num_output_files": 1, "total_output_size": 15799816, "num_input_records": 3561, "num_output_records": 3195, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:37:12.915316 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832915311, "job": 1211, "event": "table_file_deletion", "file_number": 10943}
>2018-07-11 19:37:12.918086 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531337832918083, "job": 1211, "event": "table_file_deletion", "file_number": 10941}
>2018-07-11 19:37:12.918205 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:38:07.226085 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:38:21.432584 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:38:21.588296 7f29149e4700  1 mon.magna102@0(leader).log v21906 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21906)
>2018-07-11 19:38:25.434363 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:38:25.588713 7f29149e4700  1 mon.magna102@0(leader).log v21907 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21907)
>2018-07-11 19:38:45.443038 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:38:45.598535 7f29149e4700  1 mon.magna102@0(leader).log v21908 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21908)
>2018-07-11 19:39:07.226272 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:40:01.476789 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:40:01.631548 7f29149e4700  1 mon.magna102@0(leader).log v21909 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21909)
>2018-07-11 19:40:05.478484 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:40:05.632009 7f29149e4700  1 mon.magna102@0(leader).log v21910 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21910)
>2018-07-11 19:40:07.226444 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:40:07.615815 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10945. Immutable memtables: 0.
>
>2018-07-11 19:40:07.615861 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1212] Syncing log #10942
>2018-07-11 19:40:07.632406 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.615851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:40:07.632417 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1212] Flushing memtable with next log file: 10945
>
>2018-07-11 19:40:07.632431 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338007632424, "job": 1212, "event": "flush_started", "num_memtables": 1, "num_entries": 912, "num_deletes": 251, "memory_usage": 1464408}
>2018-07-11 19:40:07.632434 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1212] Level-0 flush table #10946: started
>2018-07-11 19:40:07.682902 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338007682888, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 10946, "file_size": 1387068, "table_properties": {"data_size": 1374957, "index_size": 6169, "filter_size": 5022, "raw_key_size": 9248, "raw_average_key_size": 20, "raw_value_size": 1364300, "raw_average_value_size": 3025, "num_data_blocks": 188, "num_entries": 451, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:40:07.682913 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1212] Level-0 flush table #10946: 1387068 bytes OK
>2018-07-11 19:40:07.741294 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.699491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10946 started
>2018-07-11 19:40:07.741302 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.741257) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10946: memtable #1 done
>2018-07-11 19:40:07.741304 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.741272) EVENT_LOG_v1 {"time_micros": 1531338007741266, "job": 1212, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:40:07.741306 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.741284) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:40:07.741312 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1212] Try to delete WAL files size 1445491, prev total WAL file size 1445491, number of live WAL files 2.
>
>2018-07-11 19:40:07.741728 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:40:07.741907 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 19:40:07.741914 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 100034.6 total, 675.3 interval
>Cumulative writes: 126K writes, 592K keys, 126K commit groups, 1.0 writes per commit group, ingest: 0.97 GB, 0.01 MB/s
>Cumulative WAL: 126K writes, 126K syncs, 1.00 writes per sync, written: 0.97 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 722 writes, 3293 keys, 722 commit groups, 1.0 writes per commit group, ingest: 5.27 MB, 0.01 MB/s
>Interval WAL: 722 writes, 722 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.32 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       607    0.078       0      0
>  L1      1/1   15.07 MB   0.0      9.6     0.8      8.7       8.7     -0.0       0.0  10.7     72.5     66.3       135       604    0.223   2169K   263K
> Sum      2/2   16.39 MB   0.0      9.6     0.8      8.7       9.6      0.8       0.0  11.7     53.6     53.6       183      1211    0.151   2169K   263K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.7     54.3     55.3         1         6    0.146     10K   1248
>Uptime(secs): 100034.6 total, 100034.6 interval
>Flush(GB): cumulative 0.818, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.56 GB write, 0.10 MB/s write, 9.56 GB read, 0.10 MB/s read, 182.5 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.32 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       607    0.078       0      0
>  L1      1/1   15.07 MB   0.0      9.6     0.8      8.7       8.7     -0.0       0.0  10.7     72.5     66.3       135       604    0.223   2169K   263K
> Sum      2/2   16.39 MB   0.0      9.6     0.8      8.7       9.6      0.8       0.0  11.7     53.6     53.6       183      1211    0.151   2169K   263K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 100034.6 total, 100034.6 interval
>Flush(GB): cumulative 0.818, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.56 GB write, 0.10 MB/s write, 9.56 GB read, 0.10 MB/s read, 182.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 19:40:07.741935 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:40:07.741928) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:40:07.741939 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1213] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:40:07.741945 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1212 Base level 0, inputs: [10946(1354KB)], [10944(15MB)]
>
>2018-07-11 19:40:07.741957 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338007741948, "job": 1213, "event": "compaction_started", "files_L0": [10946], "files_L1": [10944], "score": -1, "input_data_size": 17186884}
>2018-07-11 19:40:07.933665 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1213] Generated table #10947: 3136 keys, 15309227 bytes
>2018-07-11 19:40:07.933713 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338007933686, "cf_name": "default", "job": 1213, "event": "table_file_creation", "file_number": 10947, "file_size": 15309227, "table_properties": {"data_size": 15191378, "index_size": 67531, "filter_size": 49391, "raw_key_size": 64405, "raw_average_key_size": 20, "raw_value_size": 15099052, "raw_average_value_size": 4814, "num_data_blocks": 1959, "num_entries": 3136, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:40:07.966829 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1213] Compacted 1@0 + 1@1 files to L1 => 15309227 bytes
>2018-07-11 19:40:08.000365 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:40:08.000299) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 68.1 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.1) out(14.6), read-write-amplify(23.4) write-amplify(11.0) OK, records in: 3646, records dropped: 510
>
>2018-07-11 19:40:08.000374 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:40:08.000320) EVENT_LOG_v1 {"time_micros": 1531338008000311, "job": 1213, "event": "compaction_finished", "compaction_time_micros": 224856, "output_level": 1, "num_output_files": 1, "total_output_size": 15309227, "num_input_records": 3646, "num_output_records": 3136, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:40:08.000713 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338008000709, "job": 1213, "event": "table_file_deletion", "file_number": 10946}
>2018-07-11 19:40:08.003773 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338008003770, "job": 1213, "event": "table_file_deletion", "file_number": 10944}
>2018-07-11 19:40:25.487406 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:40:25.635470 7f29149e4700  1 mon.magna102@0(leader).log v21911 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21911)
>2018-07-11 19:41:07.226614 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:41:39.519971 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:41:39.683749 7f29149e4700  1 mon.magna102@0(leader).log v21912 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21912)
>2018-07-11 19:41:45.522582 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:41:45.701065 7f29149e4700  1 mon.magna102@0(leader).log v21913 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21913)
>2018-07-11 19:42:05.531390 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:42:05.678735 7f29149e4700  1 mon.magna102@0(leader).log v21914 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21914)
>2018-07-11 19:42:07.226793 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:43:07.226972 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:43:19.564150 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:43:19.718392 7f29149e4700  1 mon.magna102@0(leader).log v21915 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21915)
>2018-07-11 19:43:25.566643 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:43:25.735615 7f29149e4700  1 mon.magna102@0(leader).log v21916 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21916)
>2018-07-11 19:43:45.575452 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:43:45.745419 7f29149e4700  1 mon.magna102@0(leader).log v21917 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21917)
>2018-07-11 19:44:07.227155 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:44:59.608217 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:44:59.760026 7f29149e4700  1 mon.magna102@0(leader).log v21918 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21918)
>2018-07-11 19:45:07.227349 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:45:11.613484 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:45:11.769325 7f29149e4700  1 mon.magna102@0(leader).log v21919 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21919)
>2018-07-11 19:45:25.619551 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:45:25.795420 7f29149e4700  1 mon.magna102@0(leader).log v21920 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21920)
>2018-07-11 19:45:32.737887 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10948. Immutable memtables: 0.
>
>2018-07-11 19:45:32.737930 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1214] Syncing log #10945
>2018-07-11 19:45:32.754356 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.737920) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:45:32.754367 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1214] Flushing memtable with next log file: 10948
>
>2018-07-11 19:45:32.754394 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338332754386, "job": 1214, "event": "flush_started", "num_memtables": 1, "num_entries": 1469, "num_deletes": 250, "memory_usage": 2689872}
>2018-07-11 19:45:32.754397 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1214] Level-0 flush table #10949: started
>2018-07-11 19:45:32.804889 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338332804875, "cf_name": "default", "job": 1214, "event": "table_file_creation", "file_number": 10949, "file_size": 1485285, "table_properties": {"data_size": 1473042, "index_size": 6081, "filter_size": 5242, "raw_key_size": 9711, "raw_average_key_size": 21, "raw_value_size": 1462377, "raw_average_value_size": 3199, "num_data_blocks": 190, "num_entries": 457, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:45:32.804900 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1214] Level-0 flush table #10949: 1485285 bytes OK
>2018-07-11 19:45:32.856115 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.821409) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10949 started
>2018-07-11 19:45:32.856123 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.856079) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10949: memtable #1 done
>2018-07-11 19:45:32.856125 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.856094) EVENT_LOG_v1 {"time_micros": 1531338332856088, "job": 1214, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:45:32.856127 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.856105) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:45:32.856147 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1214] Try to delete WAL files size 2661145, prev total WAL file size 2661145, number of live WAL files 2.
>
>2018-07-11 19:45:32.856830 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:45:32.856880 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:45:32.856873) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:45:32.856889 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1215] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:45:32.856896 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1214 Base level 0, inputs: [10949(1450KB)], [10947(14MB)]
>
>2018-07-11 19:45:32.856909 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338332856899, "job": 1215, "event": "compaction_started", "files_L0": [10949], "files_L1": [10947], "score": -1, "input_data_size": 16794512}
>2018-07-11 19:45:33.056829 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1215] Generated table #10950: 3241 keys, 16144919 bytes
>2018-07-11 19:45:33.056873 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338333056846, "cf_name": "default", "job": 1215, "event": "table_file_creation", "file_number": 10950, "file_size": 16144919, "table_properties": {"data_size": 16021562, "index_size": 70538, "filter_size": 51892, "raw_key_size": 66339, "raw_average_key_size": 20, "raw_value_size": 15925706, "raw_average_value_size": 4913, "num_data_blocks": 2055, "num_entries": 3241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:45:33.090024 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1215] Compacted 1@0 + 1@1 files to L1 => 16144919 bytes
>2018-07-11 19:45:33.123527 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:45:33.123456) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 69.3 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.6) out(15.4), read-write-amplify(22.2) write-amplify(10.9) OK, records in: 3593, records dropped: 352
>
>2018-07-11 19:45:33.123535 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:45:33.123477) EVENT_LOG_v1 {"time_micros": 1531338333123468, "job": 1215, "event": "compaction_finished", "compaction_time_micros": 233099, "output_level": 1, "num_output_files": 1, "total_output_size": 16144919, "num_input_records": 3593, "num_output_records": 3241, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:45:33.123904 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338333123900, "job": 1215, "event": "table_file_deletion", "file_number": 10949}
>2018-07-11 19:45:33.126744 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338333126741, "job": 1215, "event": "table_file_deletion", "file_number": 10947}
>2018-07-11 19:45:33.126859 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:46:07.227539 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:46:39.652221 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:46:39.796608 7f29149e4700  1 mon.magna102@0(leader).log v21921 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21921)
>2018-07-11 19:46:45.654805 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:46:45.805570 7f29149e4700  1 mon.magna102@0(leader).log v21922 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21922)
>2018-07-11 19:47:05.663516 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:47:05.815481 7f29149e4700  1 mon.magna102@0(leader).log v21923 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21923)
>2018-07-11 19:47:07.227717 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:47:59.853891 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10951. Immutable memtables: 0.
>
>2018-07-11 19:47:59.853936 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1216] Syncing log #10948
>2018-07-11 19:47:59.870460 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.853927) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:47:59.870471 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1216] Flushing memtable with next log file: 10951
>
>2018-07-11 19:47:59.870485 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338479870478, "job": 1216, "event": "flush_started", "num_memtables": 1, "num_entries": 798, "num_deletes": 251, "memory_usage": 1212464}
>2018-07-11 19:47:59.870488 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1216] Level-0 flush table #10952: started
>2018-07-11 19:47:59.920993 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338479920979, "cf_name": "default", "job": 1216, "event": "table_file_creation", "file_number": 10952, "file_size": 1146593, "table_properties": {"data_size": 1136299, "index_size": 5113, "filter_size": 4261, "raw_key_size": 8538, "raw_average_key_size": 20, "raw_value_size": 1126882, "raw_average_value_size": 2702, "num_data_blocks": 156, "num_entries": 417, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:47:59.921003 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1216] Level-0 flush table #10952: 1146593 bytes OK
>2018-07-11 19:47:59.979358 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.937550) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10952 started
>2018-07-11 19:47:59.979366 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.979322) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10952: memtable #1 done
>2018-07-11 19:47:59.979368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.979336) EVENT_LOG_v1 {"time_micros": 1531338479979330, "job": 1216, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:47:59.979370 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.979348) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:47:59.979377 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1216] Try to delete WAL files size 1195357, prev total WAL file size 1195357, number of live WAL files 2.
>
>2018-07-11 19:47:59.979773 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:47:59.979828 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:47:59.979821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:47:59.979837 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1217] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:47:59.979843 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1216 Base level 0, inputs: [10952(1119KB)], [10950(15MB)]
>
>2018-07-11 19:47:59.979855 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338479979846, "job": 1217, "event": "compaction_started", "files_L0": [10952], "files_L1": [10950], "score": -1, "input_data_size": 17291512}
>2018-07-11 19:48:00.171732 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1217] Generated table #10953: 3148 keys, 15384493 bytes
>2018-07-11 19:48:00.171776 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338480171749, "cf_name": "default", "job": 1217, "event": "table_file_creation", "file_number": 10953, "file_size": 15384493, "table_properties": {"data_size": 15266465, "index_size": 67537, "filter_size": 49564, "raw_key_size": 64630, "raw_average_key_size": 20, "raw_value_size": 15173965, "raw_average_value_size": 4820, "num_data_blocks": 1959, "num_entries": 3148, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:48:00.213246 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1217] Compacted 1@0 + 1@1 files to L1 => 15384493 bytes
>2018-07-11 19:48:00.246749 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:48:00.246683) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.1 rd, 65.9 wr, level 1, files in(1, 1) out(1) MB in(1.1, 15.4) out(14.7), read-write-amplify(28.5) write-amplify(13.4) OK, records in: 3658, records dropped: 510
>
>2018-07-11 19:48:00.246758 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:48:00.246704) EVENT_LOG_v1 {"time_micros": 1531338480246694, "job": 1217, "event": "compaction_finished", "compaction_time_micros": 233372, "output_level": 1, "num_output_files": 1, "total_output_size": 15384493, "num_input_records": 3658, "num_output_records": 3148, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:48:00.247033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338480247029, "job": 1217, "event": "table_file_deletion", "file_number": 10952}
>2018-07-11 19:48:00.250127 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338480250124, "job": 1217, "event": "table_file_deletion", "file_number": 10950}
>2018-07-11 19:48:07.227904 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:48:25.698972 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:48:25.849160 7f29149e4700  1 mon.magna102@0(leader).log v21924 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21924)
>2018-07-11 19:48:31.701555 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:48:31.849738 7f29149e4700  1 mon.magna102@0(leader).log v21925 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21925)
>2018-07-11 19:48:49.709401 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:48:49.860750 7f29149e4700  1 mon.magna102@0(leader).log v21926 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21926)
>2018-07-11 19:48:57.752729 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:48:57.869795 7f29149e4700  1 mon.magna102@0(leader).log v21927 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21927)
>2018-07-11 19:49:07.228090 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:50:05.742959 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:50:05.892422 7f29149e4700  1 mon.magna102@0(leader).log v21928 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21928)
>2018-07-11 19:50:07.228264 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:50:11.745521 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:50:11.893054 7f29149e4700  1 mon.magna102@0(leader).log v21929 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21929)
>2018-07-11 19:50:25.751542 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:50:25.902465 7f29149e4700  1 mon.magna102@0(leader).log v21930 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21930)
>2018-07-11 19:50:31.754202 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:50:31.911379 7f29149e4700  1 mon.magna102@0(leader).log v21931 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21931)
>2018-07-11 19:51:07.228440 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:51:45.786525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:51:45.934669 7f29149e4700  1 mon.magna102@0(leader).log v21932 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21932)
>2018-07-11 19:51:51.789274 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:51:51.943546 7f29149e4700  1 mon.magna102@0(leader).log v21933 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21933)
>2018-07-11 19:52:07.228629 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:52:09.797046 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:52:09.944932 7f29149e4700  1 mon.magna102@0(leader).log v21934 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21934)
>2018-07-11 19:52:17.838232 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:52:17.955118 7f29149e4700  1 mon.magna102@0(leader).log v21935 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21935)
>2018-07-11 19:52:46.957220 7f29149e4700  1 mon.magna102@0(leader).log v21936 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21936)
>2018-07-11 19:53:07.228794 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:53:25.830545 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:53:25.994703 7f29149e4700  1 mon.magna102@0(leader).log v21937 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21937)
>2018-07-11 19:53:31.833116 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:53:32.003579 7f29149e4700  1 mon.magna102@0(leader).log v21938 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21938)
>2018-07-11 19:53:49.840798 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:53:49.988296 7f29149e4700  1 mon.magna102@0(leader).log v21939 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21939)
>2018-07-11 19:53:52.938841 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10954. Immutable memtables: 0.
>
>2018-07-11 19:53:52.938886 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1218] Syncing log #10951
>2018-07-11 19:53:52.955296 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:53:52.938876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:53:52.955307 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1218] Flushing memtable with next log file: 10954
>
>2018-07-11 19:53:52.955335 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338832955327, "job": 1218, "event": "flush_started", "num_memtables": 1, "num_entries": 1615, "num_deletes": 250, "memory_usage": 3020504}
>2018-07-11 19:53:52.955338 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1218] Level-0 flush table #10955: started
>2018-07-11 19:53:53.005840 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338833005825, "cf_name": "default", "job": 1218, "event": "table_file_creation", "file_number": 10955, "file_size": 1707783, "table_properties": {"data_size": 1693979, "index_size": 6954, "filter_size": 5930, "raw_key_size": 10357, "raw_average_key_size": 21, "raw_value_size": 1682228, "raw_average_value_size": 3440, "num_data_blocks": 217, "num_entries": 489, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 19:53:53.005850 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1218] Level-0 flush table #10955: 1707783 bytes OK
>2018-07-11 19:53:53.055860 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.022355) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10955 started
>2018-07-11 19:53:53.055870 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.055811) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10955: memtable #1 done
>2018-07-11 19:53:53.055873 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.055826) EVENT_LOG_v1 {"time_micros": 1531338833055819, "job": 1218, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:53:53.055874 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.055838) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:53:53.055881 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1218] Try to delete WAL files size 2989507, prev total WAL file size 2989507, number of live WAL files 2.
>
>2018-07-11 19:53:53.056668 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:53:53.056861 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 19:53:53.056868 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 100859.9 total, 825.3 interval
>Cumulative writes: 127K writes, 596K keys, 127K commit groups, 1.0 writes per commit group, ingest: 0.98 GB, 0.01 MB/s
>Cumulative WAL: 127K writes, 127K syncs, 1.00 writes per sync, written: 0.98 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 888 writes, 3882 keys, 888 commit groups, 1.0 writes per commit group, ingest: 6.53 MB, 0.01 MB/s
>Interval WAL: 888 writes, 888 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.63 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       610    0.078       0      0
>  L1      1/1   14.67 MB   0.0      9.6     0.8      8.8       8.8     -0.0       0.0  10.7     72.5     66.3       136       607    0.223   2179K   265K
> Sum      2/2   16.30 MB   0.0      9.6     0.8      8.8       9.6      0.8       0.0  11.7     53.6     53.6       183      1217    0.151   2179K   265K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.8     54.8     54.7         1         6    0.149     10K   1372
>Uptime(secs): 100859.9 total, 100859.9 interval
>Flush(GB): cumulative 0.822, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.61 GB write, 0.10 MB/s write, 9.61 GB read, 0.10 MB/s read, 183.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.63 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       610    0.078       0      0
>  L1      1/1   14.67 MB   0.0      9.6     0.8      8.8       8.8     -0.0       0.0  10.7     72.5     66.3       136       607    0.223   2179K   265K
> Sum      2/2   16.30 MB   0.0      9.6     0.8      8.8       9.6      0.8       0.0  11.7     53.6     53.6       183      1217    0.151   2179K   265K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 100859.9 total, 100859.9 interval
>Flush(GB): cumulative 0.822, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.61 GB write, 0.10 MB/s write, 9.61 GB read, 0.10 MB/s read, 183.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 19:53:53.056889 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.056883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 19:53:53.056894 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1219] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:53:53.056899 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1218 Base level 0, inputs: [10955(1667KB)], [10953(14MB)]
>
>2018-07-11 19:53:53.056912 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338833056903, "job": 1219, "event": "compaction_started", "files_L0": [10955], "files_L1": [10953], "score": -1, "input_data_size": 17092276}
>2018-07-11 19:53:53.248258 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1219] Generated table #10956: 3299 keys, 16538364 bytes
>2018-07-11 19:53:53.248309 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338833248293, "cf_name": "default", "job": 1219, "event": "table_file_creation", "file_number": 10956, "file_size": 16538364, "table_properties": {"data_size": 16412473, "index_size": 71908, "filter_size": 53056, "raw_key_size": 67518, "raw_average_key_size": 20, "raw_value_size": 16314834, "raw_average_value_size": 4945, "num_data_blocks": 2095, "num_entries": 3299, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:53:53.289752 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1219] Compacted 1@0 + 1@1 files to L1 => 16538364 bytes
>2018-07-11 19:53:53.323261 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.323193) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.4 rd, 71.0 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.7) out(15.8), read-write-amplify(19.7) write-amplify(9.7) OK, records in: 3637, records dropped: 338
>
>2018-07-11 19:53:53.323269 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:53:53.323214) EVENT_LOG_v1 {"time_micros": 1531338833323205, "job": 1219, "event": "compaction_finished", "compaction_time_micros": 232822, "output_level": 1, "num_output_files": 1, "total_output_size": 16538364, "num_input_records": 3637, "num_output_records": 3299, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:53:53.323684 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338833323680, "job": 1219, "event": "table_file_deletion", "file_number": 10955}
>2018-07-11 19:53:53.326549 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338833326546, "job": 1219, "event": "table_file_deletion", "file_number": 10953}
>2018-07-11 19:53:53.326662 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:53:57.905370 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:53:58.024366 7f29149e4700  1 mon.magna102@0(leader).log v21940 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21940)
>2018-07-11 19:54:07.228954 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 19:55:05.874910 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:55:06.030248 7f29149e4700  1 mon.magna102@0(leader).log v21941 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21941)
>2018-07-11 19:55:07.229124 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 19:55:11.877287 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:55:12.030897 7f29149e4700  1 mon.magna102@0(leader).log v21942 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21942)
>2018-07-11 19:55:29.885142 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:55:30.058485 7f29149e4700  1 mon.magna102@0(leader).log v21943 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21943)
>2018-07-11 19:55:37.941718 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:55:38.034495 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10957. Immutable memtables: 0.
>
>2018-07-11 19:55:38.034542 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1220] Syncing log #10954
>2018-07-11 19:55:38.050955 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.034529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 19:55:38.050966 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1220] Flushing memtable with next log file: 10957
>
>2018-07-11 19:55:38.050992 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938050984, "job": 1220, "event": "flush_started", "num_memtables": 1, "num_entries": 659, "num_deletes": 251, "memory_usage": 911824}
>2018-07-11 19:55:38.050996 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1220] Level-0 flush table #10958: started
>2018-07-11 19:55:38.135001 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938134987, "cf_name": "default", "job": 1220, "event": "table_file_creation", "file_number": 10958, "file_size": 863217, "table_properties": {"data_size": 855145, "index_size": 3798, "filter_size": 3355, "raw_key_size": 7717, "raw_average_key_size": 20, "raw_value_size": 847238, "raw_average_value_size": 2241, "num_data_blocks": 116, "num_entries": 378, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 19:55:38.135012 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1220] Level-0 flush table #10958: 863217 bytes OK
>2018-07-11 19:55:38.176731 7f29149e4700  1 mon.magna102@0(leader).log v21944 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21944)
>2018-07-11 19:55:38.209986 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.176495) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10958 started
>2018-07-11 19:55:38.209994 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.209950) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10958: memtable #1 done
>2018-07-11 19:55:38.209997 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.209965) EVENT_LOG_v1 {"time_micros": 1531338938209959, "job": 1220, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 19:55:38.210011 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.209976) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 19:55:38.210018 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1220] Try to delete WAL files size 896974, prev total WAL file size 916712, number of live WAL files 2.
>
>2018-07-11 19:55:38.210300 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 19:55:38.210349 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.210342) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 19:55:38.210359 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1221] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 19:55:38.210365 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1220 Base level 0, inputs: [10958(842KB)], [10956(15MB)]
>
>2018-07-11 19:55:38.210377 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938210368, "job": 1221, "event": "compaction_started", "files_L0": [10958], "files_L1": [10956], "score": -1, "input_data_size": 17401581}
>2018-07-11 19:55:38.402408 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1221] Generated table #10959: 3167 keys, 15444654 bytes
>2018-07-11 19:55:38.402457 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938402429, "cf_name": "default", "job": 1221, "event": "table_file_creation", "file_number": 10959, "file_size": 15444654, "table_properties": {"data_size": 15326452, "index_size": 67553, "filter_size": 49722, "raw_key_size": 64988, "raw_average_key_size": 20, "raw_value_size": 15233674, "raw_average_value_size": 4810, "num_data_blocks": 1959, "num_entries": 3167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 19:55:38.435560 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1221] Compacted 1@0 + 1@1 files to L1 => 15444654 bytes
>2018-07-11 19:55:38.469088 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.469020) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.3 rd, 68.6 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.8) out(14.7), read-write-amplify(38.1) write-amplify(17.9) OK, records in: 3677, records dropped: 510
>
>2018-07-11 19:55:38.469097 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-19:55:38.469041) EVENT_LOG_v1 {"time_micros": 1531338938469032, "job": 1221, "event": "compaction_finished", "compaction_time_micros": 225167, "output_level": 1, "num_output_files": 1, "total_output_size": 15444654, "num_input_records": 3677, "num_output_records": 3167, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 19:55:38.469356 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938469351, "job": 1221, "event": "table_file_deletion", "file_number": 10958}
>2018-07-11 19:55:38.472538 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531338938472535, "job": 1221, "event": "table_file_deletion", "file_number": 10956}
>2018-07-11 19:56:07.229302 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:56:49.920326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:56:50.075594 7f29149e4700  1 mon.magna102@0(leader).log v21945 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21945)
>2018-07-11 19:56:51.921207 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:56:52.084363 7f29149e4700  1 mon.magna102@0(leader).log v21946 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21946)
>2018-07-11 19:57:07.229471 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:57:09.929047 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:57:10.102375 7f29149e4700  1 mon.magna102@0(leader).log v21947 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21947)
>2018-07-11 19:57:17.969393 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:57:18.103103 7f29149e4700  1 mon.magna102@0(leader).log v21948 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21948)
>2018-07-11 19:58:07.229655 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 19:58:25.962632 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 19:58:26.134343 7f29149e4700  1 mon.magna102@0(leader).log v21949 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21949)
>2018-07-11 19:58:31.965246 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 19:58:32.118232 7f29149e4700  1 mon.magna102@0(leader).log v21950 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21950)
>2018-07-11 19:58:49.973035 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 19:58:50.145817 7f29149e4700  1 mon.magna102@0(leader).log v21951 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21951)
>2018-07-11 19:58:58.012838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 19:58:58.129865 7f29149e4700  1 mon.magna102@0(leader).log v21952 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21952)
>2018-07-11 19:59:07.229837 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:00:06.007088 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 20:00:06.160874 7f29149e4700  1 mon.magna102@0(leader).log v21953 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21953)
>2018-07-11 20:00:07.230018 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:00:12.009509 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 20:00:12.161455 7f29149e4700  1 mon.magna102@0(leader).log v21954 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21954)
>2018-07-11 20:00:30.017279 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 20:00:30.171142 7f29149e4700  1 mon.magna102@0(leader).log v21955 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21955)
>2018-07-11 20:00:38.071870 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,6 (REQUEST_SLOW)
>2018-07-11 20:00:38.205559 7f29149e4700  1 mon.magna102@0(leader).log v21956 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21956)
>2018-07-11 20:01:07.230213 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:01:50.052883 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 20:01:50.203873 7f29149e4700  1 mon.magna102@0(leader).log v21957 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21957)
>2018-07-11 20:01:56.055544 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 20:01:56.204444 7f29149e4700  1 mon.magna102@0(leader).log v21958 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21958)
>2018-07-11 20:01:57.271248 7f29149e4700  1 mon.magna102@0(leader).log v21959 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21959)
>2018-07-11 20:02:07.198300 7f29149e4700  1 mon.magna102@0(leader).log v21960 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21960)
>2018-07-11 20:02:07.230396 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:02:13.157543 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10960. Immutable memtables: 0.
>
>2018-07-11 20:02:13.157586 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1222] Syncing log #10957
>2018-07-11 20:02:13.173982 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.157577) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:02:13.173994 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1222] Flushing memtable with next log file: 10960
>
>2018-07-11 20:02:13.174020 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333174012, "job": 1222, "event": "flush_started", "num_memtables": 1, "num_entries": 1770, "num_deletes": 250, "memory_usage": 3376184}
>2018-07-11 20:02:13.174023 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1222] Level-0 flush table #10961: started
>2018-07-11 20:02:13.232878 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333232864, "cf_name": "default", "job": 1222, "event": "table_file_creation", "file_number": 10961, "file_size": 1901796, "table_properties": {"data_size": 1886672, "index_size": 7689, "filter_size": 6515, "raw_key_size": 10838, "raw_average_key_size": 21, "raw_value_size": 1874052, "raw_average_value_size": 3653, "num_data_blocks": 240, "num_entries": 513, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:02:13.232888 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1222] Level-0 flush table #10961: 1901796 bytes OK
>2018-07-11 20:02:13.291210 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.249396) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10961 started
>2018-07-11 20:02:13.291218 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.291168) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10961: memtable #1 done
>2018-07-11 20:02:13.291221 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.291189) EVENT_LOG_v1 {"time_micros": 1531339333291176, "job": 1222, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:02:13.291223 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.291200) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:02:13.291244 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1222] Try to delete WAL files size 3342447, prev total WAL file size 3342447, number of live WAL files 2.
>
>2018-07-11 20:02:13.292075 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:02:13.292136 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.292129) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:02:13.292143 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1223] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:02:13.292150 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1222 Base level 0, inputs: [10961(1857KB)], [10959(14MB)]
>
>2018-07-11 20:02:13.292163 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333292153, "job": 1223, "event": "compaction_started", "files_L0": [10961], "files_L1": [10959], "score": -1, "input_data_size": 17346450}
>2018-07-11 20:02:13.491845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1223] Generated table #10962: 3363 keys, 16936574 bytes
>2018-07-11 20:02:13.491883 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333491868, "cf_name": "default", "job": 1223, "event": "table_file_creation", "file_number": 10962, "file_size": 16936574, "table_properties": {"data_size": 16807956, "index_size": 73416, "filter_size": 54275, "raw_key_size": 68819, "raw_average_key_size": 20, "raw_value_size": 16708348, "raw_average_value_size": 4968, "num_data_blocks": 2139, "num_entries": 3363, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:02:13.525105 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1223] Compacted 1@0 + 1@1 files to L1 => 16936574 bytes
>2018-07-11 20:02:13.566940 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.566865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.5 rd, 72.7 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.7) out(16.2), read-write-amplify(18.0) write-amplify(8.9) OK, records in: 3680, records dropped: 317
>
>2018-07-11 20:02:13.566949 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:02:13.566886) EVENT_LOG_v1 {"time_micros": 1531339333566877, "job": 1223, "event": "compaction_finished", "compaction_time_micros": 232925, "output_level": 1, "num_output_files": 1, "total_output_size": 16936574, "num_input_records": 3680, "num_output_records": 3363, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:02:13.567425 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333567420, "job": 1223, "event": "table_file_deletion", "file_number": 10961}
>2018-07-11 20:02:13.570300 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339333570284, "job": 1223, "event": "table_file_deletion", "file_number": 10959}
>2018-07-11 20:02:13.570333 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:03:07.230571 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 20:03:22.232096 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10963. Immutable memtables: 0.
>
>2018-07-11 20:03:22.232139 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1224] Syncing log #10960
>2018-07-11 20:03:22.248664 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.232130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:03:22.248676 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1224] Flushing memtable with next log file: 10963
>
>2018-07-11 20:03:22.248689 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402248682, "job": 1224, "event": "flush_started", "num_memtables": 1, "num_entries": 501, "num_deletes": 251, "memory_usage": 541208}
>2018-07-11 20:03:22.248693 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1224] Level-0 flush table #10964: started
>2018-07-11 20:03:22.290863 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402290850, "cf_name": "default", "job": 1224, "event": "table_file_creation", "file_number": 10964, "file_size": 512467, "table_properties": {"data_size": 506924, "index_size": 2352, "filter_size": 2272, "raw_key_size": 6666, "raw_average_key_size": 20, "raw_value_size": 500797, "raw_average_value_size": 1526, "num_data_blocks": 72, "num_entries": 328, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:03:22.290873 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1224] Level-0 flush table #10964: 512467 bytes OK
>2018-07-11 20:03:22.340897 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.307434) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10964 started
>2018-07-11 20:03:22.340905 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.340861) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10964: memtable #1 done
>2018-07-11 20:03:22.340908 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.340876) EVENT_LOG_v1 {"time_micros": 1531339402340870, "job": 1224, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:03:22.340910 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.340887) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:03:22.340916 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1224] Try to delete WAL files size 529236, prev total WAL file size 529236, number of live WAL files 2.
>
>2018-07-11 20:03:22.341096 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:03:22.341147 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.341140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:03:22.341156 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1225] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:03:22.341162 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1224 Base level 0, inputs: [10964(500KB)], [10962(16MB)]
>
>2018-07-11 20:03:22.341175 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402341165, "job": 1225, "event": "compaction_started", "files_L0": [10964], "files_L1": [10962], "score": -1, "input_data_size": 17449041}
>2018-07-11 20:03:22.524916 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1225] Generated table #10965: 3183 keys, 15491924 bytes
>2018-07-11 20:03:22.524964 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402524936, "cf_name": "default", "job": 1225, "event": "table_file_creation", "file_number": 10965, "file_size": 15491924, "table_properties": {"data_size": 15373573, "index_size": 67569, "filter_size": 49855, "raw_key_size": 65289, "raw_average_key_size": 20, "raw_value_size": 15280555, "raw_average_value_size": 4800, "num_data_blocks": 1959, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:03:22.558144 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1225] Compacted 1@0 + 1@1 files to L1 => 15491924 bytes
>2018-07-11 20:03:22.591663 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.591595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.4 rd, 71.4 wr, level 1, files in(1, 1) out(1) MB in(0.5, 16.2) out(14.8), read-write-amplify(64.3) write-amplify(30.2) OK, records in: 3691, records dropped: 508
>
>2018-07-11 20:03:22.591672 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:03:22.591617) EVENT_LOG_v1 {"time_micros": 1531339402591608, "job": 1225, "event": "compaction_finished", "compaction_time_micros": 216936, "output_level": 1, "num_output_files": 1, "total_output_size": 15491924, "num_input_records": 3691, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:03:22.591827 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402591822, "job": 1225, "event": "table_file_deletion", "file_number": 10964}
>2018-07-11 20:03:22.595047 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339402595044, "job": 1225, "event": "table_file_deletion", "file_number": 10962}
>2018-07-11 20:04:07.230762 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:05:07.230940 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:06:07.231123 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:07:07.231294 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:08:07.231466 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:09:07.231655 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:10:07.231845 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:10:33.379806 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10966. Immutable memtables: 0.
>
>2018-07-11 20:10:33.379850 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1226] Syncing log #10963
>2018-07-11 20:10:33.396078 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.379840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:10:33.396089 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1226] Flushing memtable with next log file: 10966
>
>2018-07-11 20:10:33.396103 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833396096, "job": 1226, "event": "flush_started", "num_memtables": 1, "num_entries": 1760, "num_deletes": 250, "memory_usage": 3224176}
>2018-07-11 20:10:33.396106 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1226] Level-0 flush table #10967: started
>2018-07-11 20:10:33.446616 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833446602, "cf_name": "default", "job": 1226, "event": "table_file_creation", "file_number": 10967, "file_size": 1658490, "table_properties": {"data_size": 1644577, "index_size": 7140, "filter_size": 5853, "raw_key_size": 10110, "raw_average_key_size": 21, "raw_value_size": 1632881, "raw_average_value_size": 3416, "num_data_blocks": 224, "num_entries": 478, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:10:33.446626 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1226] Level-0 flush table #10967: 1658490 bytes OK
>2018-07-11 20:10:33.496638 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.463169) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10967 started
>2018-07-11 20:10:33.496646 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.496602) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10967: memtable #1 done
>2018-07-11 20:10:33.496649 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.496616) EVENT_LOG_v1 {"time_micros": 1531339833496610, "job": 1226, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:10:33.496651 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.496628) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:10:33.496658 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1226] Try to delete WAL files size 3188666, prev total WAL file size 3188666, number of live WAL files 2.
>
>2018-07-11 20:10:33.497426 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:10:33.497607 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 20:10:33.497614 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 101860.4 total, 1000.4 interval
>Cumulative writes: 128K writes, 600K keys, 128K commit groups, 1.0 writes per commit group, ingest: 0.98 GB, 0.01 MB/s
>Cumulative WAL: 128K writes, 128K syncs, 1.00 writes per sync, written: 0.98 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1050 writes, 4690 keys, 1050 commit groups, 1.0 writes per commit group, ingest: 7.59 MB, 0.01 MB/s
>Interval WAL: 1050 writes, 1050 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.58 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       614    0.078       0      0
>  L1      1/1   14.77 MB   0.0      9.7     0.8      8.8       8.8     -0.0       0.0  10.7     72.5     66.3       137       611    0.223   2194K   266K
> Sum      2/2   16.36 MB   0.0      9.7     0.8      8.8       9.7      0.8       0.0  11.7     53.6     53.6       185      1225    0.151   2194K   266K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  14.0     53.5     53.6         1         8    0.154     14K   1673
>Uptime(secs): 101860.4 total, 101860.4 interval
>Flush(GB): cumulative 0.826, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.67 GB write, 0.10 MB/s write, 9.67 GB read, 0.10 MB/s read, 184.7 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.58 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       614    0.078       0      0
>  L1      1/1   14.77 MB   0.0      9.7     0.8      8.8       8.8     -0.0       0.0  10.7     72.5     66.3       137       611    0.223   2194K   266K
> Sum      2/2   16.36 MB   0.0      9.7     0.8      8.8       9.7      0.8       0.0  11.7     53.6     53.6       185      1225    0.151   2194K   266K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 101860.4 total, 101860.4 interval
>Flush(GB): cumulative 0.826, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.67 GB write, 0.10 MB/s write, 9.67 GB read, 0.10 MB/s read, 184.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 20:10:33.497634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.497628) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:10:33.497639 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1227] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:10:33.497644 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1226 Base level 0, inputs: [10967(1619KB)], [10965(14MB)]
>
>2018-07-11 20:10:33.497656 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833497647, "job": 1227, "event": "compaction_started", "files_L0": [10967], "files_L1": [10965], "score": -1, "input_data_size": 17150414}
>2018-07-11 20:10:33.697366 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1227] Generated table #10968: 3364 keys, 16867074 bytes
>2018-07-11 20:10:33.697405 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833697389, "cf_name": "default", "job": 1227, "event": "table_file_creation", "file_number": 10968, "file_size": 16867074, "table_properties": {"data_size": 16738548, "index_size": 73451, "filter_size": 54148, "raw_key_size": 68839, "raw_average_key_size": 20, "raw_value_size": 16638903, "raw_average_value_size": 4946, "num_data_blocks": 2140, "num_entries": 3364, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:10:33.738845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1227] Compacted 1@0 + 1@1 files to L1 => 16867074 bytes
>2018-07-11 20:10:33.772363 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.772298) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.1 rd, 69.9 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.8) out(16.1), read-write-amplify(20.5) write-amplify(10.2) OK, records in: 3661, records dropped: 297
>
>2018-07-11 20:10:33.772372 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:10:33.772319) EVENT_LOG_v1 {"time_micros": 1531339833772310, "job": 1227, "event": "compaction_finished", "compaction_time_micros": 241171, "output_level": 1, "num_output_files": 1, "total_output_size": 16867074, "num_input_records": 3661, "num_output_records": 3364, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:10:33.772738 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833772734, "job": 1227, "event": "table_file_deletion", "file_number": 10967}
>2018-07-11 20:10:33.775654 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339833775651, "job": 1227, "event": "table_file_deletion", "file_number": 10965}
>2018-07-11 20:10:33.775707 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:10:49.415752 7f29149e4700  1 mon.magna102@0(leader).log v21961 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21961)
>2018-07-11 20:11:07.232024 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 20:11:38.445576 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10969. Immutable memtables: 0.
>
>2018-07-11 20:11:38.445619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1228] Syncing log #10966
>2018-07-11 20:11:38.462154 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.445610) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:11:38.462166 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1228] Flushing memtable with next log file: 10969
>
>2018-07-11 20:11:38.462185 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898462172, "job": 1228, "event": "flush_started", "num_memtables": 1, "num_entries": 495, "num_deletes": 251, "memory_usage": 530824}
>2018-07-11 20:11:38.462188 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1228] Level-0 flush table #10970: started
>2018-07-11 20:11:38.504371 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898504357, "cf_name": "default", "job": 1228, "event": "table_file_creation", "file_number": 10970, "file_size": 503315, "table_properties": {"data_size": 497875, "index_size": 2283, "filter_size": 2238, "raw_key_size": 6694, "raw_average_key_size": 20, "raw_value_size": 491759, "raw_average_value_size": 1494, "num_data_blocks": 70, "num_entries": 329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:11:38.504381 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1228] Level-0 flush table #10970: 503315 bytes OK
>2018-07-11 20:11:38.554419 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.520924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10970 started
>2018-07-11 20:11:38.554427 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.554383) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10970: memtable #1 done
>2018-07-11 20:11:38.554429 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.554398) EVENT_LOG_v1 {"time_micros": 1531339898554392, "job": 1228, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:11:38.554432 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.554409) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:11:38.554437 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1228] Try to delete WAL files size 518948, prev total WAL file size 518948, number of live WAL files 2.
>
>2018-07-11 20:11:38.554619 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:11:38.554670 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.554664) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:11:38.554679 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1229] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:11:38.554686 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1228 Base level 0, inputs: [10970(491KB)], [10968(16MB)]
>
>2018-07-11 20:11:38.554698 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898554689, "job": 1229, "event": "compaction_started", "files_L0": [10970], "files_L1": [10968], "score": -1, "input_data_size": 17370389}
>2018-07-11 20:11:38.756366 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1229] Generated table #10971: 3183 keys, 15398110 bytes
>2018-07-11 20:11:38.756403 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898756387, "cf_name": "default", "job": 1229, "event": "table_file_creation", "file_number": 10971, "file_size": 15398110, "table_properties": {"data_size": 15279992, "index_size": 67532, "filter_size": 49659, "raw_key_size": 65286, "raw_average_key_size": 20, "raw_value_size": 15187000, "raw_average_value_size": 4771, "num_data_blocks": 1958, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:11:38.797889 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1229] Compacted 1@0 + 1@1 files to L1 => 15398110 bytes
>2018-07-11 20:11:38.831419 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.831346) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.4 rd, 63.3 wr, level 1, files in(1, 1) out(1) MB in(0.5, 16.1) out(14.7), read-write-amplify(65.1) write-amplify(30.6) OK, records in: 3693, records dropped: 510
>
>2018-07-11 20:11:38.831428 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:11:38.831368) EVENT_LOG_v1 {"time_micros": 1531339898831358, "job": 1229, "event": "compaction_finished", "compaction_time_micros": 243175, "output_level": 1, "num_output_files": 1, "total_output_size": 15398110, "num_input_records": 3693, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:11:38.831583 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898831579, "job": 1229, "event": "table_file_deletion", "file_number": 10970}
>2018-07-11 20:11:38.834812 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531339898834809, "job": 1229, "event": "table_file_deletion", "file_number": 10968}
>2018-07-11 20:12:07.232200 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:13:07.232392 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:14:07.232574 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:15:07.232751 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:16:07.232939 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:17:07.233103 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:18:07.233273 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:18:53.601685 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10972. Immutable memtables: 0.
>
>2018-07-11 20:18:53.601728 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1230] Syncing log #10969
>2018-07-11 20:18:53.619370 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.601718) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:18:53.619382 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1230] Flushing memtable with next log file: 10972
>
>2018-07-11 20:18:53.619408 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333619400, "job": 1230, "event": "flush_started", "num_memtables": 1, "num_entries": 1781, "num_deletes": 250, "memory_usage": 3255720}
>2018-07-11 20:18:53.619412 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1230] Level-0 flush table #10973: started
>2018-07-11 20:18:53.669901 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333669887, "cf_name": "default", "job": 1230, "event": "table_file_creation", "file_number": 10973, "file_size": 1675263, "table_properties": {"data_size": 1661232, "index_size": 7202, "filter_size": 5909, "raw_key_size": 10240, "raw_average_key_size": 21, "raw_value_size": 1649379, "raw_average_value_size": 3407, "num_data_blocks": 226, "num_entries": 484, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:18:53.669911 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1230] Level-0 flush table #10973: 1675263 bytes OK
>2018-07-11 20:18:53.719928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.686425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10973 started
>2018-07-11 20:18:53.719937 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.719892) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10973: memtable #1 done
>2018-07-11 20:18:53.719939 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.719907) EVENT_LOG_v1 {"time_micros": 1531340333719900, "job": 1230, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:18:53.719954 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.719919) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:18:53.719961 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1230] Try to delete WAL files size 3220156, prev total WAL file size 3220156, number of live WAL files 2.
>
>2018-07-11 20:18:53.720787 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:18:53.720850 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.720843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:18:53.720857 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1231] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:18:53.720863 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1230 Base level 0, inputs: [10973(1635KB)], [10971(14MB)]
>
>2018-07-11 20:18:53.720877 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333720866, "job": 1231, "event": "compaction_started", "files_L0": [10973], "files_L1": [10971], "score": -1, "input_data_size": 17073373}
>2018-07-11 20:18:53.920678 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1231] Generated table #10974: 3370 keys, 16803517 bytes
>2018-07-11 20:18:53.920727 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333920700, "cf_name": "default", "job": 1231, "event": "table_file_creation", "file_number": 10974, "file_size": 16803517, "table_properties": {"data_size": 16674991, "index_size": 73549, "filter_size": 54050, "raw_key_size": 68956, "raw_average_key_size": 20, "raw_value_size": 16575194, "raw_average_value_size": 4918, "num_data_blocks": 2143, "num_entries": 3370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:18:53.953872 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1231] Compacted 1@0 + 1@1 files to L1 => 16803517 bytes
>2018-07-11 20:18:53.987398 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.987328) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.3 rd, 72.1 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.7) out(16.0), read-write-amplify(20.2) write-amplify(10.0) OK, records in: 3667, records dropped: 297
>
>2018-07-11 20:18:53.987407 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:18:53.987349) EVENT_LOG_v1 {"time_micros": 1531340333987340, "job": 1231, "event": "compaction_finished", "compaction_time_micros": 232977, "output_level": 1, "num_output_files": 1, "total_output_size": 16803517, "num_input_records": 3667, "num_output_records": 3370, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:18:53.987814 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333987810, "job": 1231, "event": "table_file_deletion", "file_number": 10973}
>2018-07-11 20:18:53.990755 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340333990752, "job": 1231, "event": "table_file_deletion", "file_number": 10971}
>2018-07-11 20:18:53.990870 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:19:07.233463 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 20:19:17.655084 7f29149e4700  1 mon.magna102@0(leader).log v21962 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21962)
>2018-07-11 20:19:52.677207 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10975. Immutable memtables: 0.
>
>2018-07-11 20:19:52.677264 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1232] Syncing log #10972
>2018-07-11 20:19:52.693781 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.677254) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:19:52.693793 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1232] Flushing memtable with next log file: 10975
>
>2018-07-11 20:19:52.693806 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340392693799, "job": 1232, "event": "flush_started", "num_memtables": 1, "num_entries": 474, "num_deletes": 251, "memory_usage": 485912}
>2018-07-11 20:19:52.693809 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1232] Level-0 flush table #10976: started
>2018-07-11 20:19:52.735977 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340392735963, "cf_name": "default", "job": 1232, "event": "table_file_creation", "file_number": 10976, "file_size": 460302, "table_properties": {"data_size": 455197, "index_size": 2086, "filter_size": 2100, "raw_key_size": 6568, "raw_average_key_size": 20, "raw_value_size": 449304, "raw_average_value_size": 1391, "num_data_blocks": 64, "num_entries": 323, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:19:52.735987 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1232] Level-0 flush table #10976: 460302 bytes OK
>2018-07-11 20:19:52.794337 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.752531) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10976 started
>2018-07-11 20:19:52.794345 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.794302) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10976: memtable #1 done
>2018-07-11 20:19:52.794347 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.794316) EVENT_LOG_v1 {"time_micros": 1531340392794310, "job": 1232, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:19:52.794349 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.794328) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:19:52.794355 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1232] Try to delete WAL files size 474321, prev total WAL file size 474321, number of live WAL files 2.
>
>2018-07-11 20:19:52.794533 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:19:52.794585 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:19:52.794578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:19:52.794594 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1233] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:19:52.794600 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1232 Base level 0, inputs: [10976(449KB)], [10974(16MB)]
>
>2018-07-11 20:19:52.794613 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340392794603, "job": 1233, "event": "compaction_started", "files_L0": [10976], "files_L1": [10974], "score": -1, "input_data_size": 17263819}
>2018-07-11 20:19:52.978382 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1233] Generated table #10977: 3183 keys, 15298982 bytes
>2018-07-11 20:19:52.978418 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340392978402, "cf_name": "default", "job": 1233, "event": "table_file_creation", "file_number": 10977, "file_size": 15298982, "table_properties": {"data_size": 15181180, "index_size": 67430, "filter_size": 49445, "raw_key_size": 65277, "raw_average_key_size": 20, "raw_value_size": 15088263, "raw_average_value_size": 4740, "num_data_blocks": 1955, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:19:53.011568 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1233] Compacted 1@0 + 1@1 files to L1 => 15298982 bytes
>2018-07-11 20:19:53.053429 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:19:53.053363) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.6 rd, 70.5 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.0) out(14.6), read-write-amplify(70.7) write-amplify(33.2) OK, records in: 3693, records dropped: 510
>
>2018-07-11 20:19:53.053438 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:19:53.053384) EVENT_LOG_v1 {"time_micros": 1531340393053375, "job": 1233, "event": "compaction_finished", "compaction_time_micros": 216941, "output_level": 1, "num_output_files": 1, "total_output_size": 15298982, "num_input_records": 3693, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:19:53.053588 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340393053584, "job": 1233, "event": "table_file_deletion", "file_number": 10976}
>2018-07-11 20:19:53.056770 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340393056767, "job": 1233, "event": "table_file_deletion", "file_number": 10974}
>2018-07-11 20:20:07.233643 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:21:07.233802 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:22:07.233983 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:23:07.234148 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:23:07.771699 7f29149e4700  1 mon.magna102@0(leader).log v21963 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21963)
>2018-07-11 20:23:17.755886 7f29149e4700  1 mon.magna102@0(leader).log v21964 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21964)
>2018-07-11 20:24:07.234327 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:25:07.234497 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:26:07.234683 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:26:25.829677 7f29149e4700  1 mon.magna102@0(leader).log v21965 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21965)
>2018-07-11 20:27:07.234866 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:27:13.818100 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10978. Immutable memtables: 0.
>
>2018-07-11 20:27:13.818142 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1234] Syncing log #10975
>2018-07-11 20:27:13.834526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.818133) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:27:13.834537 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1234] Flushing memtable with next log file: 10978
>
>2018-07-11 20:27:13.834564 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340833834555, "job": 1234, "event": "flush_started", "num_memtables": 1, "num_entries": 1826, "num_deletes": 250, "memory_usage": 3355152}
>2018-07-11 20:27:13.834567 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1234] Level-0 flush table #10979: started
>2018-07-11 20:27:13.885102 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340833885087, "cf_name": "default", "job": 1234, "event": "table_file_creation", "file_number": 10979, "file_size": 1752401, "table_properties": {"data_size": 1737873, "index_size": 7464, "filter_size": 6144, "raw_key_size": 10534, "raw_average_key_size": 21, "raw_value_size": 1725614, "raw_average_value_size": 3465, "num_data_blocks": 234, "num_entries": 498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:27:13.885112 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1234] Level-0 flush table #10979: 1752401 bytes OK
>2018-07-11 20:27:13.935124 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.901624) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10979 started
>2018-07-11 20:27:13.935132 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.935088) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10979: memtable #1 done
>2018-07-11 20:27:13.935135 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.935103) EVENT_LOG_v1 {"time_micros": 1531340833935096, "job": 1234, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:27:13.935149 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.935115) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:27:13.935156 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1234] Try to delete WAL files size 3319273, prev total WAL file size 3319273, number of live WAL files 2.
>
>2018-07-11 20:27:13.935983 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:27:13.936167 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 20:27:13.936174 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 102860.8 total, 1000.4 interval
>Cumulative writes: 129K writes, 605K keys, 129K commit groups, 1.0 writes per commit group, ingest: 0.99 GB, 0.01 MB/s
>Cumulative WAL: 129K writes, 129K syncs, 1.00 writes per sync, written: 0.99 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1022 writes, 4576 keys, 1022 commit groups, 1.0 writes per commit group, ingest: 7.18 MB, 0.01 MB/s
>Interval WAL: 1022 writes, 1022 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.67 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       618    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.7     0.8      8.9       8.9     -0.0       0.0  10.7     72.5     66.3       137       615    0.223   2209K   268K
> Sum      2/2   16.26 MB   0.0      9.7     0.8      8.9       9.7      0.8       0.0  11.7     53.6     53.6       186      1233    0.151   2209K   268K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  15.7     55.4     55.3         1         8    0.148     14K   1614
>Uptime(secs): 102860.8 total, 102860.8 interval
>Flush(GB): cumulative 0.830, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.73 GB write, 0.10 MB/s write, 9.73 GB read, 0.10 MB/s read, 185.8 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.67 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        48       618    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.7     0.8      8.9       8.9     -0.0       0.0  10.7     72.5     66.3       137       615    0.223   2209K   268K
> Sum      2/2   16.26 MB   0.0      9.7     0.8      8.9       9.7      0.8       0.0  11.7     53.6     53.6       186      1233    0.151   2209K   268K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 102860.8 total, 102860.8 interval
>Flush(GB): cumulative 0.830, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.73 GB write, 0.10 MB/s write, 9.73 GB read, 0.10 MB/s read, 185.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 20:27:13.936208 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:27:13.936201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:27:13.936213 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1235] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:27:13.936219 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1234 Base level 0, inputs: [10979(1711KB)], [10977(14MB)]
>
>2018-07-11 20:27:13.936231 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340833936222, "job": 1235, "event": "compaction_started", "files_L0": [10979], "files_L1": [10977], "score": -1, "input_data_size": 17051383}
>2018-07-11 20:27:14.144149 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1235] Generated table #10980: 3385 keys, 16801851 bytes
>2018-07-11 20:27:14.144211 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340834144170, "cf_name": "default", "job": 1235, "event": "table_file_creation", "file_number": 10980, "file_size": 16801851, "table_properties": {"data_size": 16672925, "index_size": 73859, "filter_size": 54140, "raw_key_size": 69256, "raw_average_key_size": 20, "raw_value_size": 16572706, "raw_average_value_size": 4895, "num_data_blocks": 2152, "num_entries": 3385, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:27:14.185652 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1235] Compacted 1@0 + 1@1 files to L1 => 16801851 bytes
>2018-07-11 20:27:14.219211 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:27:14.219130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.4 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.6) out(16.0), read-write-amplify(19.3) write-amplify(9.6) OK, records in: 3681, records dropped: 296
>
>2018-07-11 20:27:14.219220 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:27:14.219151) EVENT_LOG_v1 {"time_micros": 1531340834219141, "job": 1235, "event": "compaction_finished", "compaction_time_micros": 249403, "output_level": 1, "num_output_files": 1, "total_output_size": 16801851, "num_input_records": 3681, "num_output_records": 3385, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:27:14.219642 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340834219638, "job": 1235, "event": "table_file_deletion", "file_number": 10979}
>2018-07-11 20:27:14.222607 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340834222605, "job": 1235, "event": "table_file_deletion", "file_number": 10977}
>2018-07-11 20:27:14.222724 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:28:02.866483 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10981. Immutable memtables: 0.
>
>2018-07-11 20:28:02.866525 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1236] Syncing log #10978
>2018-07-11 20:28:02.883007 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.866515) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:28:02.883019 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1236] Flushing memtable with next log file: 10981
>
>2018-07-11 20:28:02.883032 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340882883025, "job": 1236, "event": "flush_started", "num_memtables": 1, "num_entries": 431, "num_deletes": 251, "memory_usage": 392208}
>2018-07-11 20:28:02.883035 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1236] Level-0 flush table #10982: started
>2018-07-11 20:28:02.925233 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340882925220, "cf_name": "default", "job": 1236, "event": "table_file_creation", "file_number": 10982, "file_size": 370137, "table_properties": {"data_size": 365709, "index_size": 1693, "filter_size": 1816, "raw_key_size": 6246, "raw_average_key_size": 20, "raw_value_size": 360347, "raw_average_value_size": 1169, "num_data_blocks": 52, "num_entries": 308, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:28:02.925243 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1236] Level-0 flush table #10982: 370137 bytes OK
>2018-07-11 20:28:02.975288 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.941797) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10982 started
>2018-07-11 20:28:02.975296 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.975253) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10982: memtable #1 done
>2018-07-11 20:28:02.975299 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.975268) EVENT_LOG_v1 {"time_micros": 1531340882975262, "job": 1236, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:28:02.975301 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.975279) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:28:02.975307 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1236] Try to delete WAL files size 381546, prev total WAL file size 381546, number of live WAL files 2.
>
>2018-07-11 20:28:02.975462 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:28:02.975513 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:28:02.975506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:28:02.975522 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1237] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:28:02.975528 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1236 Base level 0, inputs: [10982(361KB)], [10980(16MB)]
>
>2018-07-11 20:28:02.975540 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340882975531, "job": 1237, "event": "compaction_started", "files_L0": [10982], "files_L1": [10980], "score": -1, "input_data_size": 17171988}
>2018-07-11 20:28:03.159321 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1237] Generated table #10983: 3185 keys, 15289560 bytes
>2018-07-11 20:28:03.159352 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340883159338, "cf_name": "default", "job": 1237, "event": "table_file_creation", "file_number": 10983, "file_size": 15289560, "table_properties": {"data_size": 15171890, "index_size": 67331, "filter_size": 49412, "raw_key_size": 65306, "raw_average_key_size": 20, "raw_value_size": 15079021, "raw_average_value_size": 4734, "num_data_blocks": 1952, "num_entries": 3185, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:28:03.200845 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1237] Compacted 1@0 + 1@1 files to L1 => 15289560 bytes
>2018-07-11 20:28:03.234378 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:28:03.234310) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.2 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.0) out(14.6), read-write-amplify(87.7) write-amplify(41.3) OK, records in: 3693, records dropped: 508
>
>2018-07-11 20:28:03.234386 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:28:03.234332) EVENT_LOG_v1 {"time_micros": 1531340883234323, "job": 1237, "event": "compaction_finished", "compaction_time_micros": 225288, "output_level": 1, "num_output_files": 1, "total_output_size": 15289560, "num_input_records": 3693, "num_output_records": 3185, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:28:03.234519 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340883234515, "job": 1237, "event": "table_file_deletion", "file_number": 10982}
>2018-07-11 20:28:03.237711 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531340883237708, "job": 1237, "event": "table_file_deletion", "file_number": 10980}
>2018-07-11 20:28:07.235039 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:29:07.235211 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:30:07.235387 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:31:07.235568 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:32:07.235741 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:32:27.033568 7f29149e4700  1 mon.magna102@0(leader).log v21966 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21966)
>2018-07-11 20:33:07.235906 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:34:07.236074 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:35:07.236244 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:35:34.032485 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10984. Immutable memtables: 0.
>
>2018-07-11 20:35:34.032528 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1238] Syncing log #10981
>2018-07-11 20:35:34.048893 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.032519) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:35:34.048904 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1238] Flushing memtable with next log file: 10984
>
>2018-07-11 20:35:34.048931 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334048923, "job": 1238, "event": "flush_started", "num_memtables": 1, "num_entries": 1838, "num_deletes": 250, "memory_usage": 3397064}
>2018-07-11 20:35:34.048935 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1238] Level-0 flush table #10985: started
>2018-07-11 20:35:34.107767 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334107753, "cf_name": "default", "job": 1238, "event": "table_file_creation", "file_number": 10985, "file_size": 1754780, "table_properties": {"data_size": 1740226, "index_size": 7489, "filter_size": 6145, "raw_key_size": 10422, "raw_average_key_size": 21, "raw_value_size": 1728043, "raw_average_value_size": 3505, "num_data_blocks": 235, "num_entries": 493, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:35:34.107777 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1238] Level-0 flush table #10985: 1754780 bytes OK
>2018-07-11 20:35:34.157748 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.124261) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10985 started
>2018-07-11 20:35:34.157756 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.157712) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10985: memtable #1 done
>2018-07-11 20:35:34.157758 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.157726) EVENT_LOG_v1 {"time_micros": 1531341334157720, "job": 1238, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:35:34.157760 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.157738) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:35:34.157780 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1238] Try to delete WAL files size 3361617, prev total WAL file size 3361617, number of live WAL files 2.
>
>2018-07-11 20:35:34.158677 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:35:34.158740 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.158733) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:35:34.158747 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1239] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:35:34.158753 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1238 Base level 0, inputs: [10985(1713KB)], [10983(14MB)]
>
>2018-07-11 20:35:34.158766 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334158756, "job": 1239, "event": "compaction_started", "files_L0": [10985], "files_L1": [10983], "score": -1, "input_data_size": 17044340}
>2018-07-11 20:35:34.358506 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1239] Generated table #10986: 3389 keys, 16828789 bytes
>2018-07-11 20:35:34.358550 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334358523, "cf_name": "default", "job": 1239, "event": "table_file_creation", "file_number": 10986, "file_size": 16828789, "table_properties": {"data_size": 16699680, "index_size": 73960, "filter_size": 54222, "raw_key_size": 69337, "raw_average_key_size": 20, "raw_value_size": 16599335, "raw_average_value_size": 4898, "num_data_blocks": 2155, "num_entries": 3389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:35:34.391677 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1239] Compacted 1@0 + 1@1 files to L1 => 16828789 bytes
>2018-07-11 20:35:34.425222 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.425144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.2 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.6) out(16.0), read-write-amplify(19.3) write-amplify(9.6) OK, records in: 3678, records dropped: 289
>
>2018-07-11 20:35:34.425231 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:35:34.425166) EVENT_LOG_v1 {"time_micros": 1531341334425156, "job": 1239, "event": "compaction_finished", "compaction_time_micros": 232895, "output_level": 1, "num_output_files": 1, "total_output_size": 16828789, "num_input_records": 3678, "num_output_records": 3389, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:35:34.425654 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334425650, "job": 1239, "event": "table_file_deletion", "file_number": 10985}
>2018-07-11 20:35:34.428641 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341334428638, "job": 1239, "event": "table_file_deletion", "file_number": 10983}
>2018-07-11 20:35:34.428754 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:36:07.236418 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 20:36:19.080463 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10987. Immutable memtables: 0.
>
>2018-07-11 20:36:19.080504 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1240] Syncing log #10984
>2018-07-11 20:36:19.097072 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.080495) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:36:19.097084 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1240] Flushing memtable with next log file: 10987
>
>2018-07-11 20:36:19.097098 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379097090, "job": 1240, "event": "flush_started", "num_memtables": 1, "num_entries": 417, "num_deletes": 251, "memory_usage": 364136}
>2018-07-11 20:36:19.097100 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1240] Level-0 flush table #10988: started
>2018-07-11 20:36:19.139266 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379139253, "cf_name": "default", "job": 1240, "event": "table_file_creation", "file_number": 10988, "file_size": 343576, "table_properties": {"data_size": 339368, "index_size": 1561, "filter_size": 1728, "raw_key_size": 6162, "raw_average_key_size": 20, "raw_value_size": 334153, "raw_average_value_size": 1099, "num_data_blocks": 48, "num_entries": 304, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:36:19.139277 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1240] Level-0 flush table #10988: 343576 bytes OK
>2018-07-11 20:36:19.189307 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.155842) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10988 started
>2018-07-11 20:36:19.189315 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.189272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10988: memtable #1 done
>2018-07-11 20:36:19.189317 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.189286) EVENT_LOG_v1 {"time_micros": 1531341379189280, "job": 1240, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:36:19.189319 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.189298) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:36:19.189325 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1240] Try to delete WAL files size 353904, prev total WAL file size 353904, number of live WAL files 2.
>
>2018-07-11 20:36:19.189474 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:36:19.189525 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.189518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:36:19.189534 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1241] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:36:19.189540 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1240 Base level 0, inputs: [10988(335KB)], [10986(16MB)]
>
>2018-07-11 20:36:19.189553 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379189543, "job": 1241, "event": "compaction_started", "files_L0": [10988], "files_L1": [10986], "score": -1, "input_data_size": 17172365}
>2018-07-11 20:36:19.389901 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1241] Generated table #10989: 3185 keys, 15296576 bytes
>2018-07-11 20:36:19.389945 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379389918, "cf_name": "default", "job": 1241, "event": "table_file_creation", "file_number": 10989, "file_size": 15296576, "table_properties": {"data_size": 15178891, "index_size": 67331, "filter_size": 49427, "raw_key_size": 65303, "raw_average_key_size": 20, "raw_value_size": 15086024, "raw_average_value_size": 4736, "num_data_blocks": 1952, "num_entries": 3185, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:36:19.423211 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1241] Compacted 1@0 + 1@1 files to L1 => 15296576 bytes
>2018-07-11 20:36:19.456720 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.456653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.5 rd, 65.5 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.0) out(14.6), read-write-amplify(94.5) write-amplify(44.5) OK, records in: 3693, records dropped: 508
>
>2018-07-11 20:36:19.456728 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:36:19.456674) EVENT_LOG_v1 {"time_micros": 1531341379456665, "job": 1241, "event": "compaction_finished", "compaction_time_micros": 233642, "output_level": 1, "num_output_files": 1, "total_output_size": 15296576, "num_input_records": 3693, "num_output_records": 3185, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:36:19.456857 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379456852, "job": 1241, "event": "table_file_deletion", "file_number": 10988}
>2018-07-11 20:36:19.459966 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341379459963, "job": 1241, "event": "table_file_deletion", "file_number": 10986}
>2018-07-11 20:37:07.236601 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:38:07.236774 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:39:07.236959 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:40:07.237138 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:41:07.237332 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:42:07.237513 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:43:07.237689 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:43:17.043341 7f29149e4700  1 mon.magna102@0(leader).log v21967 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21967)
>2018-07-11 20:43:54.271499 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10990. Immutable memtables: 0.
>
>2018-07-11 20:43:54.271542 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1242] Syncing log #10987
>2018-07-11 20:43:54.289152 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.271533) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:43:54.289164 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1242] Flushing memtable with next log file: 10990
>
>2018-07-11 20:43:54.289178 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834289170, "job": 1242, "event": "flush_started", "num_memtables": 1, "num_entries": 1852, "num_deletes": 250, "memory_usage": 3436232}
>2018-07-11 20:43:54.289188 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1242] Level-0 flush table #10991: started
>2018-07-11 20:43:54.339689 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834339675, "cf_name": "default", "job": 1242, "event": "table_file_creation", "file_number": 10991, "file_size": 1774169, "table_properties": {"data_size": 1759492, "index_size": 7554, "filter_size": 6203, "raw_key_size": 10462, "raw_average_key_size": 21, "raw_value_size": 1747237, "raw_average_value_size": 3529, "num_data_blocks": 237, "num_entries": 495, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:43:54.339699 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1242] Level-0 flush table #10991: 1774169 bytes OK
>2018-07-11 20:43:54.398084 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.356235) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10991 started
>2018-07-11 20:43:54.398092 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.398048) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10991: memtable #1 done
>2018-07-11 20:43:54.398094 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.398062) EVENT_LOG_v1 {"time_micros": 1531341834398056, "job": 1242, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:43:54.398096 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.398074) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:43:54.398103 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1242] Try to delete WAL files size 3400779, prev total WAL file size 3400779, number of live WAL files 2.
>
>2018-07-11 20:43:54.398939 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:43:54.399120 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 20:43:54.399127 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 103861.3 total, 1000.5 interval
>Cumulative writes: 130K writes, 610K keys, 130K commit groups, 1.0 writes per commit group, ingest: 1.00 GB, 0.01 MB/s
>Cumulative WAL: 130K writes, 130K syncs, 1.00 writes per sync, written: 1.00 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1012 writes, 4538 keys, 1012 commit groups, 1.0 writes per commit group, ingest: 7.15 MB, 0.01 MB/s
>Interval WAL: 1012 writes, 1012 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.69 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        49       622    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.8     0.8      9.0       9.0     -0.0       0.0  10.8     72.5     66.3       138       619    0.224   2224K   270K
> Sum      2/2   16.28 MB   0.0      9.8     0.8      9.0       9.8      0.8       0.0  11.7     53.6     53.6       187      1241    0.151   2224K   270K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.1     54.3     54.4         1         8    0.150     14K   1601
>Uptime(secs): 103861.3 total, 103861.3 interval
>Flush(GB): cumulative 0.834, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.80 GB write, 0.10 MB/s write, 9.80 GB read, 0.10 MB/s read, 187.0 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.69 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.6        49       622    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.8     0.8      9.0       9.0     -0.0       0.0  10.8     72.5     66.3       138       619    0.224   2224K   270K
> Sum      2/2   16.28 MB   0.0      9.8     0.8      9.0       9.8      0.8       0.0  11.7     53.6     53.6       187      1241    0.151   2224K   270K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 103861.3 total, 103861.3 interval
>Flush(GB): cumulative 0.834, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.80 GB write, 0.10 MB/s write, 9.80 GB read, 0.10 MB/s read, 187.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 20:43:54.399150 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.399143) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:43:54.399155 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1243] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:43:54.399160 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1242 Base level 0, inputs: [10991(1732KB)], [10989(14MB)]
>
>2018-07-11 20:43:54.399172 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834399163, "job": 1243, "event": "compaction_started", "files_L0": [10991], "files_L1": [10989], "score": -1, "input_data_size": 17070745}
>2018-07-11 20:43:54.598801 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1243] Generated table #10992: 3393 keys, 16867539 bytes
>2018-07-11 20:43:54.598845 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834598818, "cf_name": "default", "job": 1243, "event": "table_file_creation", "file_number": 10992, "file_size": 16867539, "table_properties": {"data_size": 16738179, "index_size": 74097, "filter_size": 54336, "raw_key_size": 69418, "raw_average_key_size": 20, "raw_value_size": 16637684, "raw_average_value_size": 4903, "num_data_blocks": 2159, "num_entries": 3393, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:43:54.631972 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1243] Compacted 1@0 + 1@1 files to L1 => 16867539 bytes
>2018-07-11 20:43:54.673840 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.673765) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.3 rd, 72.5 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.6) out(16.1), read-write-amplify(19.1) write-amplify(9.5) OK, records in: 3680, records dropped: 287
>
>2018-07-11 20:43:54.673849 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:43:54.673789) EVENT_LOG_v1 {"time_micros": 1531341834673777, "job": 1243, "event": "compaction_finished", "compaction_time_micros": 232783, "output_level": 1, "num_output_files": 1, "total_output_size": 16867539, "num_input_records": 3680, "num_output_records": 3393, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:43:54.674276 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834674271, "job": 1243, "event": "table_file_deletion", "file_number": 10991}
>2018-07-11 20:43:54.677274 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341834677271, "job": 1243, "event": "table_file_deletion", "file_number": 10989}
>2018-07-11 20:43:54.677304 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:44:06.283581 7f29149e4700  1 mon.magna102@0(leader).log v21968 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21968)
>2018-07-11 20:44:07.237871 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 20:44:33.302672 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10993. Immutable memtables: 0.
>
>2018-07-11 20:44:33.302713 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1244] Syncing log #10990
>2018-07-11 20:44:33.319222 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.302703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:44:33.319233 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1244] Flushing memtable with next log file: 10993
>
>2018-07-11 20:44:33.319260 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873319252, "job": 1244, "event": "flush_started", "num_memtables": 1, "num_entries": 404, "num_deletes": 251, "memory_usage": 335304}
>2018-07-11 20:44:33.319263 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1244] Level-0 flush table #10994: started
>2018-07-11 20:44:33.353107 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873353094, "cf_name": "default", "job": 1244, "event": "table_file_creation", "file_number": 10994, "file_size": 316583, "table_properties": {"data_size": 312598, "index_size": 1426, "filter_size": 1640, "raw_key_size": 6148, "raw_average_key_size": 20, "raw_value_size": 307470, "raw_average_value_size": 1014, "num_data_blocks": 44, "num_entries": 303, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:44:33.353117 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1244] Level-0 flush table #10994: 316583 bytes OK
>2018-07-11 20:44:33.403135 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.369640) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10994 started
>2018-07-11 20:44:33.403144 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.403101) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10994: memtable #1 done
>2018-07-11 20:44:33.403146 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.403116) EVENT_LOG_v1 {"time_micros": 1531341873403110, "job": 1244, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:44:33.403161 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.403127) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:44:33.403166 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1244] Try to delete WAL files size 325247, prev total WAL file size 325247, number of live WAL files 2.
>
>2018-07-11 20:44:33.403323 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:44:33.403370 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.403364) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:44:33.403380 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1245] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:44:33.403386 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1244 Base level 0, inputs: [10994(309KB)], [10992(16MB)]
>
>2018-07-11 20:44:33.403399 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873403389, "job": 1245, "event": "compaction_started", "files_L0": [10994], "files_L1": [10992], "score": -1, "input_data_size": 17184122}
>2018-07-11 20:44:33.587212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1245] Generated table #10995: 3186 keys, 15305066 bytes
>2018-07-11 20:44:33.587262 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873587233, "cf_name": "default", "job": 1245, "event": "table_file_creation", "file_number": 10995, "file_size": 15305066, "table_properties": {"data_size": 15187361, "index_size": 67330, "filter_size": 49448, "raw_key_size": 65319, "raw_average_key_size": 20, "raw_value_size": 15094483, "raw_average_value_size": 4737, "num_data_blocks": 1952, "num_entries": 3186, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:44:33.620311 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1245] Compacted 1@0 + 1@1 files to L1 => 15305066 bytes
>2018-07-11 20:44:33.653886 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.653819) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.1) out(14.6), read-write-amplify(102.6) write-amplify(48.3) OK, records in: 3696, records dropped: 510
>
>2018-07-11 20:44:33.653895 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:44:33.653841) EVENT_LOG_v1 {"time_micros": 1531341873653832, "job": 1245, "event": "compaction_finished", "compaction_time_micros": 216898, "output_level": 1, "num_output_files": 1, "total_output_size": 15305066, "num_input_records": 3696, "num_output_records": 3186, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:44:33.654037 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873654033, "job": 1245, "event": "table_file_deletion", "file_number": 10994}
>2018-07-11 20:44:33.657156 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531341873657153, "job": 1245, "event": "table_file_deletion", "file_number": 10992}
>2018-07-11 20:45:07.238056 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:46:07.246429 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:47:07.310151 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:48:07.310328 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:49:07.328052 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:50:07.350352 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:51:07.372440 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:51:26.949008 7f29149e4700  1 mon.magna102@0(leader).log v21969 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21969)
>2018-07-11 20:52:07.403142 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:52:14.479149 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10996. Immutable memtables: 0.
>
>2018-07-11 20:52:14.479197 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1246] Syncing log #10993
>2018-07-11 20:52:14.495603 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.479187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:52:14.495615 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1246] Flushing memtable with next log file: 10996
>
>2018-07-11 20:52:14.495641 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334495632, "job": 1246, "event": "flush_started", "num_memtables": 1, "num_entries": 1873, "num_deletes": 250, "memory_usage": 3465528}
>2018-07-11 20:52:14.495644 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1246] Level-0 flush table #10997: started
>2018-07-11 20:52:14.546136 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334546121, "cf_name": "default", "job": 1246, "event": "table_file_creation", "file_number": 10997, "file_size": 1788642, "table_properties": {"data_size": 1773817, "index_size": 7647, "filter_size": 6258, "raw_key_size": 10522, "raw_average_key_size": 21, "raw_value_size": 1761448, "raw_average_value_size": 3537, "num_data_blocks": 240, "num_entries": 498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 20:52:14.546146 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1246] Level-0 flush table #10997: 1788642 bytes OK
>2018-07-11 20:52:14.596152 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.562671) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #10997 started
>2018-07-11 20:52:14.596160 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.596114) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #10997: memtable #1 done
>2018-07-11 20:52:14.596162 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.596129) EVENT_LOG_v1 {"time_micros": 1531342334596123, "job": 1246, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:52:14.596177 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.596141) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:52:14.596191 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1246] Try to delete WAL files size 3429304, prev total WAL file size 3429304, number of live WAL files 2.
>
>2018-07-11 20:52:14.597017 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:52:14.597079 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.597072) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 20:52:14.597085 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1247] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:52:14.597091 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1246 Base level 0, inputs: [10997(1746KB)], [10995(14MB)]
>
>2018-07-11 20:52:14.597105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334597095, "job": 1247, "event": "compaction_started", "files_L0": [10997], "files_L1": [10995], "score": -1, "input_data_size": 17093708}
>2018-07-11 20:52:14.805234 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1247] Generated table #10998: 3400 keys, 16912054 bytes
>2018-07-11 20:52:14.805285 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334805268, "cf_name": "default", "job": 1247, "event": "table_file_creation", "file_number": 10998, "file_size": 16912054, "table_properties": {"data_size": 16782346, "index_size": 74298, "filter_size": 54483, "raw_key_size": 69560, "raw_average_key_size": 20, "raw_value_size": 16681612, "raw_average_value_size": 4906, "num_data_blocks": 2165, "num_entries": 3400, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:52:14.838392 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1247] Compacted 1@0 + 1@1 files to L1 => 16912054 bytes
>2018-07-11 20:52:14.871917 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.871848) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 70.8 rd, 70.1 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.6) out(16.1), read-write-amplify(19.0) write-amplify(9.5) OK, records in: 3684, records dropped: 284
>
>2018-07-11 20:52:14.871926 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:14.871869) EVENT_LOG_v1 {"time_micros": 1531342334871860, "job": 1247, "event": "compaction_finished", "compaction_time_micros": 241272, "output_level": 1, "num_output_files": 1, "total_output_size": 16912054, "num_input_records": 3684, "num_output_records": 3400, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:52:14.872359 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334872355, "job": 1247, "event": "table_file_deletion", "file_number": 10997}
>2018-07-11 20:52:14.875418 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342334875415, "job": 1247, "event": "table_file_deletion", "file_number": 10995}
>2018-07-11 20:52:14.875450 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:52:49.526553 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #10999. Immutable memtables: 0.
>
>2018-07-11 20:52:49.526594 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1248] Syncing log #10996
>2018-07-11 20:52:49.543133 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.526585) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 20:52:49.543144 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1248] Flushing memtable with next log file: 10999
>
>2018-07-11 20:52:49.543158 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369543151, "job": 1248, "event": "flush_started", "num_memtables": 1, "num_entries": 382, "num_deletes": 251, "memory_usage": 289456}
>2018-07-11 20:52:49.543161 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1248] Level-0 flush table #11000: started
>2018-07-11 20:52:49.585338 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369585325, "cf_name": "default", "job": 1248, "event": "table_file_creation", "file_number": 11000, "file_size": 272061, "table_properties": {"data_size": 268411, "index_size": 1233, "filter_size": 1498, "raw_key_size": 5952, "raw_average_key_size": 20, "raw_value_size": 263576, "raw_average_value_size": 896, "num_data_blocks": 38, "num_entries": 294, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 20:52:49.585348 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1248] Level-0 flush table #11000: 272061 bytes OK
>2018-07-11 20:52:49.635405 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.601916) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11000 started
>2018-07-11 20:52:49.635413 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.635370) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11000: memtable #1 done
>2018-07-11 20:52:49.635415 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.635385) EVENT_LOG_v1 {"time_micros": 1531342369635378, "job": 1248, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 20:52:49.635417 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.635396) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 20:52:49.635423 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1248] Try to delete WAL files size 279707, prev total WAL file size 279707, number of live WAL files 2.
>
>2018-07-11 20:52:49.635561 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 20:52:49.635615 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.635608) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 20:52:49.635624 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1249] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 20:52:49.635630 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1248 Base level 0, inputs: [11000(265KB)], [10998(16MB)]
>
>2018-07-11 20:52:49.635643 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369635633, "job": 1249, "event": "compaction_started", "files_L0": [11000], "files_L1": [10998], "score": -1, "input_data_size": 17184115}
>2018-07-11 20:52:49.819437 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1249] Generated table #11001: 3186 keys, 15297703 bytes
>2018-07-11 20:52:49.819473 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369819457, "cf_name": "default", "job": 1249, "event": "table_file_creation", "file_number": 11001, "file_size": 15297703, "table_properties": {"data_size": 15180054, "index_size": 67297, "filter_size": 49425, "raw_key_size": 65316, "raw_average_key_size": 20, "raw_value_size": 15087202, "raw_average_value_size": 4735, "num_data_blocks": 1951, "num_entries": 3186, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 20:52:49.852650 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1249] Compacted 1@0 + 1@1 files to L1 => 15297703 bytes
>2018-07-11 20:52:49.886188 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.886108) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 70.5 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.1) out(14.6), read-write-amplify(119.4) write-amplify(56.2) OK, records in: 3694, records dropped: 508
>
>2018-07-11 20:52:49.886197 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-20:52:49.886130) EVENT_LOG_v1 {"time_micros": 1531342369886120, "job": 1249, "event": "compaction_finished", "compaction_time_micros": 216991, "output_level": 1, "num_output_files": 1, "total_output_size": 15297703, "num_input_records": 3694, "num_output_records": 3186, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 20:52:49.886314 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369886310, "job": 1249, "event": "table_file_deletion", "file_number": 11000}
>2018-07-11 20:52:49.889385 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342369889383, "job": 1249, "event": "table_file_deletion", "file_number": 10998}
>2018-07-11 20:53:07.429333 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:54:07.451486 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:55:07.481773 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:56:07.502583 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:57:07.541217 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:58:07.554964 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 20:59:07.585413 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:00:07.616112 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:00:34.710325 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11002. Immutable memtables: 0.
>
>2018-07-11 21:00:34.710369 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1250] Syncing log #10999
>2018-07-11 21:00:34.726751 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.710359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:00:34.726762 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1250] Flushing memtable with next log file: 11002
>
>2018-07-11 21:00:34.726788 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342834726780, "job": 1250, "event": "flush_started", "num_memtables": 1, "num_entries": 1879, "num_deletes": 250, "memory_usage": 3494672}
>2018-07-11 21:00:34.726791 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1250] Level-0 flush table #11003: started
>2018-07-11 21:00:34.785623 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342834785609, "cf_name": "default", "job": 1250, "event": "table_file_creation", "file_number": 11003, "file_size": 1795122, "table_properties": {"data_size": 1780245, "index_size": 7683, "filter_size": 6274, "raw_key_size": 10450, "raw_average_key_size": 21, "raw_value_size": 1767920, "raw_average_value_size": 3571, "num_data_blocks": 241, "num_entries": 495, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:00:34.785634 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1250] Level-0 flush table #11003: 1795122 bytes OK
>2018-07-11 21:00:34.835632 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.802153) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11003 started
>2018-07-11 21:00:34.835640 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.835596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11003: memtable #1 done
>2018-07-11 21:00:34.835643 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.835610) EVENT_LOG_v1 {"time_micros": 1531342834835604, "job": 1250, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:00:34.835658 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.835623) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:00:34.835664 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1250] Try to delete WAL files size 3458963, prev total WAL file size 3458963, number of live WAL files 2.
>
>2018-07-11 21:00:34.836558 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:00:34.836751 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 21:00:34.836758 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 104861.7 total, 1000.4 interval
>Cumulative writes: 131K writes, 614K keys, 131K commit groups, 1.0 writes per commit group, ingest: 1.00 GB, 0.01 MB/s
>Cumulative WAL: 131K writes, 131K syncs, 1.00 writes per sync, written: 1.00 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1012 writes, 4538 keys, 1012 commit groups, 1.0 writes per commit group, ingest: 7.15 MB, 0.01 MB/s
>Interval WAL: 1012 writes, 1012 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       626    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.9     0.8      9.0       9.0     -0.0       0.0  10.8     72.5     66.3       139       623    0.224   2238K   271K
> Sum      2/2   16.30 MB   0.0      9.9     0.8      9.0       9.9      0.8       0.0  11.8     53.7     53.7       188      1249    0.151   2238K   271K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.4     56.4     56.4         1         8    0.145     14K   1589
>Uptime(secs): 104861.7 total, 104861.7 interval
>Flush(GB): cumulative 0.838, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.86 GB write, 0.10 MB/s write, 9.86 GB read, 0.10 MB/s read, 188.2 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       626    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.9     0.8      9.0       9.0     -0.0       0.0  10.8     72.5     66.3       139       623    0.224   2238K   271K
> Sum      2/2   16.30 MB   0.0      9.9     0.8      9.0       9.9      0.8       0.0  11.8     53.7     53.7       188      1249    0.151   2238K   271K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 104861.7 total, 104861.7 interval
>Flush(GB): cumulative 0.838, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.86 GB write, 0.10 MB/s write, 9.86 GB read, 0.10 MB/s read, 188.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 21:00:34.836779 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:00:34.836772) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:00:34.836784 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1251] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:00:34.836789 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1250 Base level 0, inputs: [11003(1753KB)], [11001(14MB)]
>
>2018-07-11 21:00:34.836802 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342834836792, "job": 1251, "event": "compaction_started", "files_L0": [11003], "files_L1": [11001], "score": -1, "input_data_size": 17092825}
>2018-07-11 21:00:35.053016 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1251] Generated table #11004: 3401 keys, 16923693 bytes
>2018-07-11 21:00:35.053064 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342835053037, "cf_name": "default", "job": 1251, "event": "table_file_creation", "file_number": 11004, "file_size": 16923693, "table_properties": {"data_size": 16793918, "index_size": 74332, "filter_size": 54516, "raw_key_size": 69580, "raw_average_key_size": 20, "raw_value_size": 16693147, "raw_average_value_size": 4908, "num_data_blocks": 2166, "num_entries": 3401, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:00:35.086215 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1251] Compacted 1@0 + 1@1 files to L1 => 16923693 bytes
>2018-07-11 21:00:35.128060 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:00:35.127993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.5 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.6) out(16.1), read-write-amplify(18.9) write-amplify(9.4) OK, records in: 3681, records dropped: 280
>
>2018-07-11 21:00:35.128069 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:00:35.128014) EVENT_LOG_v1 {"time_micros": 1531342835128004, "job": 1251, "event": "compaction_finished", "compaction_time_micros": 249396, "output_level": 1, "num_output_files": 1, "total_output_size": 16923693, "num_input_records": 3681, "num_output_records": 3401, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:00:35.128499 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342835128495, "job": 1251, "event": "table_file_deletion", "file_number": 11003}
>2018-07-11 21:00:35.131506 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342835131503, "job": 1251, "event": "table_file_deletion", "file_number": 11001}
>2018-07-11 21:00:35.131575 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:01:07.640696 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 21:01:07.749566 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11005. Immutable memtables: 0.
>
>2018-07-11 21:01:07.749610 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1252] Syncing log #11002
>2018-07-11 21:01:07.766165 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.749601) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:01:07.766177 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1252] Flushing memtable with next log file: 11005
>
>2018-07-11 21:01:07.766211 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342867766203, "job": 1252, "event": "flush_started", "num_memtables": 1, "num_entries": 375, "num_deletes": 251, "memory_usage": 274536}
>2018-07-11 21:01:07.766214 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1252] Level-0 flush table #11006: started
>2018-07-11 21:01:07.808371 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342867808357, "cf_name": "default", "job": 1252, "event": "table_file_creation", "file_number": 11006, "file_size": 257699, "table_properties": {"data_size": 254157, "index_size": 1167, "filter_size": 1456, "raw_key_size": 5910, "raw_average_key_size": 20, "raw_value_size": 249403, "raw_average_value_size": 854, "num_data_blocks": 36, "num_entries": 292, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:01:07.808381 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1252] Level-0 flush table #11006: 257699 bytes OK
>2018-07-11 21:01:07.858396 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.824912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11006 started
>2018-07-11 21:01:07.858404 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.858361) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11006: memtable #1 done
>2018-07-11 21:01:07.858407 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.858375) EVENT_LOG_v1 {"time_micros": 1531342867858369, "job": 1252, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:01:07.858409 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.858387) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:01:07.858428 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1252] Try to delete WAL files size 264810, prev total WAL file size 264810, number of live WAL files 2.
>
>2018-07-11 21:01:07.858574 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:01:07.858625 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:01:07.858617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:01:07.858634 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1253] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:01:07.858640 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1252 Base level 0, inputs: [11006(251KB)], [11004(16MB)]
>
>2018-07-11 21:01:07.858654 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342867858644, "job": 1253, "event": "compaction_started", "files_L0": [11006], "files_L1": [11004], "score": -1, "input_data_size": 17181392}
>2018-07-11 21:01:08.050762 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1253] Generated table #11007: 3185 keys, 15289708 bytes
>2018-07-11 21:01:08.050810 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342868050783, "cf_name": "default", "job": 1253, "event": "table_file_creation", "file_number": 11007, "file_size": 15289708, "table_properties": {"data_size": 15172115, "index_size": 67262, "filter_size": 49404, "raw_key_size": 65294, "raw_average_key_size": 20, "raw_value_size": 15079302, "raw_average_value_size": 4734, "num_data_blocks": 1950, "num_entries": 3185, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:01:08.083949 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1253] Compacted 1@0 + 1@1 files to L1 => 15289708 bytes
>2018-07-11 21:01:08.117487 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:01:08.117420) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.3 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.1) out(14.6), read-write-amplify(126.0) write-amplify(59.3) OK, records in: 3693, records dropped: 508
>
>2018-07-11 21:01:08.117496 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:01:08.117442) EVENT_LOG_v1 {"time_micros": 1531342868117432, "job": 1253, "event": "compaction_finished", "compaction_time_micros": 225278, "output_level": 1, "num_output_files": 1, "total_output_size": 15289708, "num_input_records": 3693, "num_output_records": 3185, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:01:08.117629 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342868117624, "job": 1253, "event": "table_file_deletion", "file_number": 11006}
>2018-07-11 21:01:08.120786 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531342868120783, "job": 1253, "event": "table_file_deletion", "file_number": 11004}
>2018-07-11 21:02:07.665091 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:02:36.958880 7f29149e4700  1 mon.magna102@0(leader).log v21970 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21970)
>2018-07-11 21:02:38.125922 7f29149e4700  1 mon.magna102@0(leader).log v21971 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21971)
>2018-07-11 21:02:48.777976 7f29149e4700  1 mon.magna102@0(leader).log v21972 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21972)
>2018-07-11 21:03:07.687656 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:03:23.130567 7f29149e4700  1 mon.magna102@0(leader).log v21973 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21973)
>2018-07-11 21:04:07.718238 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:04:46.821097 7f29149e4700  1 mon.magna102@0(leader).log v21974 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21974)
>2018-07-11 21:05:07.740395 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:06:07.769530 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:06:35.489471 7f29149e4700  1 mon.magna102@0(leader).log v21975 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21975)
>2018-07-11 21:07:07.791727 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:08:07.813723 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:08:54.910442 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11008. Immutable memtables: 0.
>
>2018-07-11 21:08:54.910486 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1254] Syncing log #11005
>2018-07-11 21:08:54.926890 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:08:54.910476) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:08:54.926901 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1254] Flushing memtable with next log file: 11008
>
>2018-07-11 21:08:54.926915 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343334926907, "job": 1254, "event": "flush_started", "num_memtables": 1, "num_entries": 1934, "num_deletes": 250, "memory_usage": 3594200}
>2018-07-11 21:08:54.926918 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1254] Level-0 flush table #11009: started
>2018-07-11 21:08:54.985776 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343334985762, "cf_name": "default", "job": 1254, "event": "table_file_creation", "file_number": 11009, "file_size": 1892685, "table_properties": {"data_size": 1877101, "index_size": 8077, "filter_size": 6587, "raw_key_size": 10887, "raw_average_key_size": 21, "raw_value_size": 1864173, "raw_average_value_size": 3612, "num_data_blocks": 253, "num_entries": 516, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:08:54.985787 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1254] Level-0 flush table #11009: 1892685 bytes OK
>2018-07-11 21:08:55.035813 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.002321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11009 started
>2018-07-11 21:08:55.035822 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.035777) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11009: memtable #1 done
>2018-07-11 21:08:55.035824 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.035791) EVENT_LOG_v1 {"time_micros": 1531343335035785, "job": 1254, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:08:55.035826 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.035803) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:08:55.035832 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1254] Try to delete WAL files size 3556456, prev total WAL file size 3556456, number of live WAL files 2.
>
>2018-07-11 21:08:55.036691 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:08:55.036744 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.036737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:08:55.036753 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1255] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:08:55.036759 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1254 Base level 0, inputs: [11009(1848KB)], [11007(14MB)]
>
>2018-07-11 21:08:55.036772 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343335036762, "job": 1255, "event": "compaction_started", "files_L0": [11009], "files_L1": [11007], "score": -1, "input_data_size": 17182393}
>2018-07-11 21:08:55.228241 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1255] Generated table #11010: 3420 keys, 17020130 bytes
>2018-07-11 21:08:55.228293 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343335228264, "cf_name": "default", "job": 1255, "event": "table_file_creation", "file_number": 11010, "file_size": 17020130, "table_properties": {"data_size": 16889595, "index_size": 74776, "filter_size": 54832, "raw_key_size": 69966, "raw_average_key_size": 20, "raw_value_size": 16788238, "raw_average_value_size": 4908, "num_data_blocks": 2179, "num_entries": 3420, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:08:55.261395 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1255] Compacted 1@0 + 1@1 files to L1 => 17020130 bytes
>2018-07-11 21:08:55.294915 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.294840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.5 rd, 75.8 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.6) out(16.2), read-write-amplify(18.1) write-amplify(9.0) OK, records in: 3701, records dropped: 281
>
>2018-07-11 21:08:55.294924 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:08:55.294862) EVENT_LOG_v1 {"time_micros": 1531343335294852, "job": 1255, "event": "compaction_finished", "compaction_time_micros": 224602, "output_level": 1, "num_output_files": 1, "total_output_size": 17020130, "num_input_records": 3701, "num_output_records": 3420, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:08:55.295343 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343335295338, "job": 1255, "event": "table_file_deletion", "file_number": 11009}
>2018-07-11 21:08:55.298369 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343335298353, "job": 1255, "event": "table_file_deletion", "file_number": 11007}
>2018-07-11 21:08:55.298402 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:09:07.846215 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 21:09:13.947114 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11011. Immutable memtables: 0.
>
>2018-07-11 21:09:13.947153 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1256] Syncing log #11008
>2018-07-11 21:09:13.963733 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:09:13.947144) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:09:13.963745 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1256] Flushing memtable with next log file: 11011
>
>2018-07-11 21:09:13.963758 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343353963751, "job": 1256, "event": "flush_started", "num_memtables": 1, "num_entries": 326, "num_deletes": 251, "memory_usage": 168784}
>2018-07-11 21:09:13.963761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1256] Level-0 flush table #11012: started
>2018-07-11 21:09:13.997601 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343353997588, "cf_name": "default", "job": 1256, "event": "table_file_creation", "file_number": 11012, "file_size": 156658, "table_properties": {"data_size": 153899, "index_size": 706, "filter_size": 1134, "raw_key_size": 5616, "raw_average_key_size": 20, "raw_value_size": 149672, "raw_average_value_size": 538, "num_data_blocks": 22, "num_entries": 278, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:09:13.997611 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1256] Level-0 flush table #11012: 156658 bytes OK
>2018-07-11 21:09:14.047628 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.014172) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11012 started
>2018-07-11 21:09:14.047637 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.047595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11012: memtable #1 done
>2018-07-11 21:09:14.047640 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.047609) EVENT_LOG_v1 {"time_micros": 1531343354047603, "job": 1256, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:09:14.047644 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.047620) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:09:14.047649 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1256] Try to delete WAL files size 160011, prev total WAL file size 160011, number of live WAL files 2.
>
>2018-07-11 21:09:14.047760 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:09:14.047810 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.047804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:09:14.047819 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1257] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:09:14.047825 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1256 Base level 0, inputs: [11012(152KB)], [11010(16MB)]
>
>2018-07-11 21:09:14.047837 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343354047828, "job": 1257, "event": "compaction_started", "files_L0": [11012], "files_L1": [11010], "score": -1, "input_data_size": 17176788}
>2018-07-11 21:09:14.231650 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1257] Generated table #11013: 3190 keys, 15293624 bytes
>2018-07-11 21:09:14.231695 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343354231667, "cf_name": "default", "job": 1257, "event": "table_file_creation", "file_number": 11013, "file_size": 15293624, "table_properties": {"data_size": 15176046, "index_size": 67233, "filter_size": 49418, "raw_key_size": 65386, "raw_average_key_size": 20, "raw_value_size": 15083179, "raw_average_value_size": 4728, "num_data_blocks": 1949, "num_entries": 3190, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:09:14.264865 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1257] Compacted 1@0 + 1@1 files to L1 => 15293624 bytes
>2018-07-11 21:09:14.306712 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.306646) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.2 rd, 70.5 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.2) out(14.6), read-write-amplify(207.3) write-amplify(97.6) OK, records in: 3698, records dropped: 508
>
>2018-07-11 21:09:14.306721 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:09:14.306667) EVENT_LOG_v1 {"time_micros": 1531343354306658, "job": 1257, "event": "compaction_finished", "compaction_time_micros": 217014, "output_level": 1, "num_output_files": 1, "total_output_size": 15293624, "num_input_records": 3698, "num_output_records": 3190, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:09:14.306821 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343354306816, "job": 1257, "event": "table_file_deletion", "file_number": 11012}
>2018-07-11 21:09:14.309921 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343354309919, "job": 1257, "event": "table_file_deletion", "file_number": 11010}
>2018-07-11 21:10:07.879020 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:11:07.901233 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:12:07.923104 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:13:07.953563 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:14:07.974378 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:15:08.004616 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:16:08.026756 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:17:08.057017 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:17:15.133083 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11014. Immutable memtables: 0.
>
>2018-07-11 21:17:15.133126 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1258] Syncing log #11011
>2018-07-11 21:17:15.149509 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.133117) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:17:15.149521 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1258] Flushing memtable with next log file: 11014
>
>2018-07-11 21:17:15.149534 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835149527, "job": 1258, "event": "flush_started", "num_memtables": 1, "num_entries": 1935, "num_deletes": 250, "memory_usage": 3596120}
>2018-07-11 21:17:15.149537 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1258] Level-0 flush table #11015: started
>2018-07-11 21:17:15.208383 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835208368, "cf_name": "default", "job": 1258, "event": "table_file_creation", "file_number": 11015, "file_size": 1844877, "table_properties": {"data_size": 1829577, "index_size": 7938, "filter_size": 6442, "raw_key_size": 10610, "raw_average_key_size": 21, "raw_value_size": 1816956, "raw_average_value_size": 3612, "num_data_blocks": 249, "num_entries": 503, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:17:15.208394 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1258] Level-0 flush table #11015: 1844877 bytes OK
>2018-07-11 21:17:15.258443 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.224941) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11015 started
>2018-07-11 21:17:15.258451 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.258406) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11015: memtable #1 done
>2018-07-11 21:17:15.258453 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.258420) EVENT_LOG_v1 {"time_micros": 1531343835258414, "job": 1258, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:17:15.258455 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.258432) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:17:15.258462 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1258] Try to delete WAL files size 3557891, prev total WAL file size 3557891, number of live WAL files 2.
>
>2018-07-11 21:17:15.259284 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:17:15.259464 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 21:17:15.259470 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 105862.1 total, 1000.4 interval
>Cumulative writes: 132K writes, 619K keys, 132K commit groups, 1.0 writes per commit group, ingest: 1.01 GB, 0.01 MB/s
>Cumulative WAL: 132K writes, 132K syncs, 1.00 writes per sync, written: 1.01 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1020 writes, 4570 keys, 1020 commit groups, 1.0 writes per commit group, ingest: 7.19 MB, 0.01 MB/s
>Interval WAL: 1020 writes, 1020 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.76 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       630    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.9     0.8      9.1       9.1     -0.0       0.0  10.8     72.5     66.3       140       627    0.224   2253K   273K
> Sum      2/2   16.34 MB   0.0      9.9     0.8      9.1       9.9      0.8       0.0  11.8     53.7     53.7       189      1257    0.151   2253K   273K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  16.5     55.6     55.7         1         8    0.147     14K   1577
>Uptime(secs): 105862.1 total, 105862.1 interval
>Flush(GB): cumulative 0.842, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.93 GB write, 0.10 MB/s write, 9.93 GB read, 0.10 MB/s read, 189.4 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.76 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       630    0.078       0      0
>  L1      1/1   14.59 MB   0.0      9.9     0.8      9.1       9.1     -0.0       0.0  10.8     72.5     66.3       140       627    0.224   2253K   273K
> Sum      2/2   16.34 MB   0.0      9.9     0.8      9.1       9.9      0.8       0.0  11.8     53.7     53.7       189      1257    0.151   2253K   273K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 105862.1 total, 105862.1 interval
>Flush(GB): cumulative 0.842, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.93 GB write, 0.10 MB/s write, 9.93 GB read, 0.10 MB/s read, 189.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 21:17:15.259491 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.259484) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:17:15.259495 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1259] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:17:15.259501 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1258 Base level 0, inputs: [11015(1801KB)], [11013(14MB)]
>
>2018-07-11 21:17:15.259513 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835259504, "job": 1259, "event": "compaction_started", "files_L0": [11015], "files_L1": [11013], "score": -1, "input_data_size": 17138501}
>2018-07-11 21:17:15.459140 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1259] Generated table #11016: 3421 keys, 17024633 bytes
>2018-07-11 21:17:15.459197 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835459156, "cf_name": "default", "job": 1259, "event": "table_file_creation", "file_number": 11016, "file_size": 17024633, "table_properties": {"data_size": 16894046, "index_size": 74811, "filter_size": 54849, "raw_key_size": 69986, "raw_average_key_size": 20, "raw_value_size": 16792652, "raw_average_value_size": 4908, "num_data_blocks": 2180, "num_entries": 3421, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:17:15.492318 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1259] Compacted 1@0 + 1@1 files to L1 => 17024633 bytes
>2018-07-11 21:17:15.525855 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.525789) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.6 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.6) out(16.2), read-write-amplify(18.5) write-amplify(9.2) OK, records in: 3693, records dropped: 272
>
>2018-07-11 21:17:15.525864 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:15.525810) EVENT_LOG_v1 {"time_micros": 1531343835525801, "job": 1259, "event": "compaction_finished", "compaction_time_micros": 232789, "output_level": 1, "num_output_files": 1, "total_output_size": 17024633, "num_input_records": 3693, "num_output_records": 3421, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:17:15.526258 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835526254, "job": 1259, "event": "table_file_deletion", "file_number": 11015}
>2018-07-11 21:17:15.529247 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343835529244, "job": 1259, "event": "table_file_deletion", "file_number": 11013}
>2018-07-11 21:17:15.529274 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:17:32.161297 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11017. Immutable memtables: 0.
>
>2018-07-11 21:17:32.161340 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1260] Syncing log #11014
>2018-07-11 21:17:32.177868 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.161331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:17:32.177879 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1260] Flushing memtable with next log file: 11017
>
>2018-07-11 21:17:32.177892 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852177885, "job": 1260, "event": "flush_started", "num_memtables": 1, "num_entries": 319, "num_deletes": 251, "memory_usage": 153872}
>2018-07-11 21:17:32.177895 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1260] Level-0 flush table #11018: started
>2018-07-11 21:17:32.211659 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852211646, "cf_name": "default", "job": 1260, "event": "table_file_creation", "file_number": 11018, "file_size": 142420, "table_properties": {"data_size": 139773, "index_size": 640, "filter_size": 1088, "raw_key_size": 5574, "raw_average_key_size": 20, "raw_value_size": 135627, "raw_average_value_size": 491, "num_data_blocks": 20, "num_entries": 276, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:17:32.211669 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1260] Level-0 flush table #11018: 142420 bytes OK
>2018-07-11 21:17:32.263029 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.228312) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11018 started
>2018-07-11 21:17:32.263037 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.262995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11018: memtable #1 done
>2018-07-11 21:17:32.263040 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.263010) EVENT_LOG_v1 {"time_micros": 1531343852263004, "job": 1260, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:17:32.263042 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.263021) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:17:32.263047 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1260] Try to delete WAL files size 145242, prev total WAL file size 145242, number of live WAL files 2.
>
>2018-07-11 21:17:32.263156 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:17:32.263216 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.263210) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:17:32.263225 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1261] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:17:32.263231 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1260 Base level 0, inputs: [11018(139KB)], [11016(16MB)]
>
>2018-07-11 21:17:32.263243 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852263234, "job": 1261, "event": "compaction_started", "files_L0": [11018], "files_L1": [11016], "score": -1, "input_data_size": 17167053}
>2018-07-11 21:17:32.455416 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1261] Generated table #11019: 3189 keys, 15276503 bytes
>2018-07-11 21:17:32.455445 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852455431, "cf_name": "default", "job": 1261, "event": "table_file_creation", "file_number": 11019, "file_size": 15276503, "table_properties": {"data_size": 15159005, "index_size": 67198, "filter_size": 49373, "raw_key_size": 65364, "raw_average_key_size": 20, "raw_value_size": 15066177, "raw_average_value_size": 4724, "num_data_blocks": 1948, "num_entries": 3189, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:17:32.488601 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1261] Compacted 1@0 + 1@1 files to L1 => 15276503 bytes
>2018-07-11 21:17:32.522123 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.522054) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.2 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.2) out(14.6), read-write-amplify(227.8) write-amplify(107.3) OK, records in: 3697, records dropped: 508
>
>2018-07-11 21:17:32.522132 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:17:32.522076) EVENT_LOG_v1 {"time_micros": 1531343852522067, "job": 1261, "event": "compaction_finished", "compaction_time_micros": 225341, "output_level": 1, "num_output_files": 1, "total_output_size": 15276503, "num_input_records": 3697, "num_output_records": 3189, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:17:32.522238 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852522233, "job": 1261, "event": "table_file_deletion", "file_number": 11018}
>2018-07-11 21:17:32.525342 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531343852525339, "job": 1261, "event": "table_file_deletion", "file_number": 11016}
>2018-07-11 21:18:08.083243 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:19:08.105349 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:19:47.233214 7f29149e4700  1 mon.magna102@0(leader).log v21976 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21976)
>2018-07-11 21:20:08.136157 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:21:08.150219 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:22:08.187595 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:23:08.209547 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:24:08.240039 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:25:08.262011 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:25:35.347720 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11020. Immutable memtables: 0.
>
>2018-07-11 21:25:35.347765 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1262] Syncing log #11017
>2018-07-11 21:25:35.364157 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.347756) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:25:35.364168 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1262] Flushing memtable with next log file: 11020
>
>2018-07-11 21:25:35.364189 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335364175, "job": 1262, "event": "flush_started", "num_memtables": 1, "num_entries": 1957, "num_deletes": 250, "memory_usage": 3628672}
>2018-07-11 21:25:35.364194 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1262] Level-0 flush table #11021: started
>2018-07-11 21:25:35.423106 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335423091, "cf_name": "default", "job": 1262, "event": "table_file_creation", "file_number": 11021, "file_size": 1869569, "table_properties": {"data_size": 1854131, "index_size": 7998, "filter_size": 6520, "raw_key_size": 10832, "raw_average_key_size": 21, "raw_value_size": 1841271, "raw_average_value_size": 3589, "num_data_blocks": 251, "num_entries": 513, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:25:35.423116 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1262] Level-0 flush table #11021: 1869569 bytes OK
>2018-07-11 21:25:35.481438 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.439599) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11021 started
>2018-07-11 21:25:35.481446 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.481401) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11021: memtable #1 done
>2018-07-11 21:25:35.481448 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.481416) EVENT_LOG_v1 {"time_micros": 1531344335481410, "job": 1262, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:25:35.481450 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.481428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:25:35.481456 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1262] Try to delete WAL files size 3589998, prev total WAL file size 3589998, number of live WAL files 2.
>
>2018-07-11 21:25:35.482293 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:25:35.482347 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.482340) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:25:35.482356 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1263] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:25:35.482362 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1262 Base level 0, inputs: [11021(1825KB)], [11019(14MB)]
>
>2018-07-11 21:25:35.482375 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335482365, "job": 1263, "event": "compaction_started", "files_L0": [11021], "files_L1": [11019], "score": -1, "input_data_size": 17146072}
>2018-07-11 21:25:35.707139 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1263] Generated table #11022: 3427 keys, 17038955 bytes
>2018-07-11 21:25:35.707196 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335707156, "cf_name": "default", "job": 1263, "event": "table_file_creation", "file_number": 11022, "file_size": 17038955, "table_properties": {"data_size": 16908180, "index_size": 74937, "filter_size": 54911, "raw_key_size": 70103, "raw_average_key_size": 20, "raw_value_size": 16806615, "raw_average_value_size": 4904, "num_data_blocks": 2184, "num_entries": 3427, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:25:35.740332 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1263] Compacted 1@0 + 1@1 files to L1 => 17038955 bytes
>2018-07-11 21:25:35.782216 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.782126) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 66.5 rd, 66.1 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.6) out(16.2), read-write-amplify(18.3) write-amplify(9.1) OK, records in: 3702, records dropped: 275
>
>2018-07-11 21:25:35.782226 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:35.782147) EVENT_LOG_v1 {"time_micros": 1531344335782137, "job": 1263, "event": "compaction_finished", "compaction_time_micros": 257941, "output_level": 1, "num_output_files": 1, "total_output_size": 17038955, "num_input_records": 3702, "num_output_records": 3427, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:25:35.782639 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335782635, "job": 1263, "event": "table_file_deletion", "file_number": 11021}
>2018-07-11 21:25:35.785672 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344335785669, "job": 1263, "event": "table_file_deletion", "file_number": 11019}
>2018-07-11 21:25:35.785727 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:25:46.385158 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11023. Immutable memtables: 0.
>
>2018-07-11 21:25:46.385266 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1264] Syncing log #11020
>2018-07-11 21:25:46.401742 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.385257) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:25:46.401753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1264] Flushing memtable with next log file: 11023
>
>2018-07-11 21:25:46.401766 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346401759, "job": 1264, "event": "flush_started", "num_memtables": 1, "num_entries": 298, "num_deletes": 251, "memory_usage": 109160}
>2018-07-11 21:25:46.401769 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1264] Level-0 flush table #11024: started
>2018-07-11 21:25:46.443954 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346443942, "cf_name": "default", "job": 1264, "event": "table_file_creation", "file_number": 11024, "file_size": 99728, "table_properties": {"data_size": 97415, "index_size": 444, "filter_size": 950, "raw_key_size": 5448, "raw_average_key_size": 20, "raw_value_size": 93492, "raw_average_value_size": 346, "num_data_blocks": 14, "num_entries": 270, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:25:46.443964 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1264] Level-0 flush table #11024: 99728 bytes OK
>2018-07-11 21:25:46.494000 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.460538) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11024 started
>2018-07-11 21:25:46.494008 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.493966) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11024: memtable #1 done
>2018-07-11 21:25:46.494011 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.493980) EVENT_LOG_v1 {"time_micros": 1531344346493974, "job": 1264, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:25:46.494013 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.493991) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:25:46.494017 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1264] Try to delete WAL files size 100935, prev total WAL file size 100935, number of live WAL files 2.
>
>2018-07-11 21:25:46.494119 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:25:46.494167 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.494162) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:25:46.494176 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1265] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:25:46.494192 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1264 Base level 0, inputs: [11024(97KB)], [11022(16MB)]
>
>2018-07-11 21:25:46.494219 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346494208, "job": 1265, "event": "compaction_started", "files_L0": [11024], "files_L1": [11022], "score": -1, "input_data_size": 17138683}
>2018-07-11 21:25:46.669705 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1265] Generated table #11025: 3189 keys, 15251725 bytes
>2018-07-11 21:25:46.669749 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346669722, "cf_name": "default", "job": 1265, "event": "table_file_creation", "file_number": 11025, "file_size": 15251725, "table_properties": {"data_size": 15134371, "index_size": 67122, "filter_size": 49305, "raw_key_size": 65355, "raw_average_key_size": 20, "raw_value_size": 15041600, "raw_average_value_size": 4716, "num_data_blocks": 1946, "num_entries": 3189, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:25:46.702892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1265] Compacted 1@0 + 1@1 files to L1 => 15251725 bytes
>2018-07-11 21:25:46.752193 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.752111) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.1 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.2) out(14.5), read-write-amplify(324.8) write-amplify(152.9) OK, records in: 3697, records dropped: 508
>
>2018-07-11 21:25:46.752201 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:25:46.752133) EVENT_LOG_v1 {"time_micros": 1531344346752123, "job": 1265, "event": "compaction_finished", "compaction_time_micros": 208656, "output_level": 1, "num_output_files": 1, "total_output_size": 15251725, "num_input_records": 3697, "num_output_records": 3189, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:25:46.752291 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346752287, "job": 1265, "event": "table_file_deletion", "file_number": 11024}
>2018-07-11 21:25:46.755389 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344346755386, "job": 1265, "event": "table_file_deletion", "file_number": 11022}
>2018-07-11 21:26:08.288237 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:27:08.310379 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:28:08.332428 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:29:08.363149 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:30:08.392290 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:31:08.414454 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:32:08.453194 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:33:08.466908 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:33:55.571820 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11026. Immutable memtables: 0.
>
>2018-07-11 21:33:55.571865 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1266] Syncing log #11023
>2018-07-11 21:33:55.588318 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.571856) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:33:55.588330 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1266] Flushing memtable with next log file: 11026
>
>2018-07-11 21:33:55.588344 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835588336, "job": 1266, "event": "flush_started", "num_memtables": 1, "num_entries": 1963, "num_deletes": 250, "memory_usage": 3655336}
>2018-07-11 21:33:55.588346 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1266] Level-0 flush table #11027: started
>2018-07-11 21:33:55.638870 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835638856, "cf_name": "default", "job": 1266, "event": "table_file_creation", "file_number": 11027, "file_size": 1874696, "table_properties": {"data_size": 1859177, "index_size": 8065, "filter_size": 6534, "raw_key_size": 10690, "raw_average_key_size": 21, "raw_value_size": 1846408, "raw_average_value_size": 3641, "num_data_blocks": 253, "num_entries": 507, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:33:55.638880 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1266] Level-0 flush table #11027: 1874696 bytes OK
>2018-07-11 21:33:55.688922 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.655300) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11027 started
>2018-07-11 21:33:55.688930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.688884) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11027: memtable #1 done
>2018-07-11 21:33:55.688933 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.688899) EVENT_LOG_v1 {"time_micros": 1531344835688893, "job": 1266, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:33:55.688935 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.688912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:33:55.688941 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1266] Try to delete WAL files size 3616967, prev total WAL file size 3616967, number of live WAL files 2.
>
>2018-07-11 21:33:55.689812 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:33:55.689992 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 21:33:55.689999 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 106862.5 total, 1000.4 interval
>Cumulative writes: 133K writes, 623K keys, 133K commit groups, 1.0 writes per commit group, ingest: 1.02 GB, 0.01 MB/s
>Cumulative WAL: 133K writes, 133K syncs, 1.00 writes per sync, written: 1.02 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1012 writes, 4537 keys, 1012 commit groups, 1.0 writes per commit group, ingest: 7.11 MB, 0.01 MB/s
>Interval WAL: 1012 writes, 1012 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.79 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       634    0.078       0      0
>  L1      1/1   14.55 MB   0.0     10.0     0.8      9.1       9.1     -0.0       0.0  10.8     72.5     66.4       141       631    0.224   2268K   274K
> Sum      2/2   16.33 MB   0.0     10.0     0.8      9.1      10.0      0.8       0.0  11.8     53.7     53.7       191      1265    0.151   2268K   274K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  17.2     55.6     55.6         1         8    0.147     14K   1563
>Uptime(secs): 106862.5 total, 106862.5 interval
>Flush(GB): cumulative 0.846, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.99 GB write, 0.10 MB/s write, 9.99 GB read, 0.10 MB/s read, 190.6 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.79 MB   0.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     17.5        49       634    0.078       0      0
>  L1      1/1   14.55 MB   0.0     10.0     0.8      9.1       9.1     -0.0       0.0  10.8     72.5     66.4       141       631    0.224   2268K   274K
> Sum      2/2   16.33 MB   0.0     10.0     0.8      9.1      10.0      0.8       0.0  11.8     53.7     53.7       191      1265    0.151   2268K   274K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 106862.5 total, 106862.5 interval
>Flush(GB): cumulative 0.846, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 9.99 GB write, 0.10 MB/s write, 9.99 GB read, 0.10 MB/s read, 190.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 21:33:55.690019 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.690013) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:33:55.690024 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1267] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:33:55.690031 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1266 Base level 0, inputs: [11027(1830KB)], [11025(14MB)]
>
>2018-07-11 21:33:55.690043 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835690034, "job": 1267, "event": "compaction_started", "files_L0": [11027], "files_L1": [11025], "score": -1, "input_data_size": 17126421}
>2018-07-11 21:33:55.881298 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1267] Generated table #11028: 3428 keys, 17039676 bytes
>2018-07-11 21:33:55.881337 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835881321, "cf_name": "default", "job": 1267, "event": "table_file_creation", "file_number": 11028, "file_size": 17039676, "table_properties": {"data_size": 16908855, "index_size": 74970, "filter_size": 54924, "raw_key_size": 70123, "raw_average_key_size": 20, "raw_value_size": 16807253, "raw_average_value_size": 4902, "num_data_blocks": 2185, "num_entries": 3428, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:33:55.914466 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1267] Compacted 1@0 + 1@1 files to L1 => 17039676 bytes
>2018-07-11 21:33:55.948008 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.947939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.3 rd, 75.9 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.5) out(16.3), read-write-amplify(18.2) write-amplify(9.1) OK, records in: 3696, records dropped: 268
>
>2018-07-11 21:33:55.948017 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:33:55.947960) EVENT_LOG_v1 {"time_micros": 1531344835947950, "job": 1267, "event": "compaction_finished", "compaction_time_micros": 224409, "output_level": 1, "num_output_files": 1, "total_output_size": 17039676, "num_input_records": 3696, "num_output_records": 3428, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:33:55.948421 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835948417, "job": 1267, "event": "table_file_deletion", "file_number": 11027}
>2018-07-11 21:33:55.951402 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344835951399, "job": 1267, "event": "table_file_deletion", "file_number": 11025}
>2018-07-11 21:33:55.951431 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:34:04.591145 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11029. Immutable memtables: 0.
>
>2018-07-11 21:34:04.591191 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1268] Syncing log #11026
>2018-07-11 21:34:04.607771 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.591176) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:34:04.607783 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1268] Flushing memtable with next log file: 11029
>
>2018-07-11 21:34:04.607796 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844607789, "job": 1268, "event": "flush_started", "num_memtables": 1, "num_entries": 291, "num_deletes": 251, "memory_usage": 94240}
>2018-07-11 21:34:04.607799 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1268] Level-0 flush table #11030: started
>2018-07-11 21:34:04.649963 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844649951, "cf_name": "default", "job": 1268, "event": "table_file_creation", "file_number": 11030, "file_size": 85491, "table_properties": {"data_size": 83290, "index_size": 378, "filter_size": 904, "raw_key_size": 5406, "raw_average_key_size": 20, "raw_value_size": 79447, "raw_average_value_size": 296, "num_data_blocks": 12, "num_entries": 268, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:34:04.649973 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1268] Level-0 flush table #11030: 85491 bytes OK
>2018-07-11 21:34:04.708360 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.666558) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11030 started
>2018-07-11 21:34:04.708368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.708325) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11030: memtable #1 done
>2018-07-11 21:34:04.708371 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.708341) EVENT_LOG_v1 {"time_micros": 1531344844708335, "job": 1268, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:34:04.708373 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.708352) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:34:04.708378 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1268] Try to delete WAL files size 86166, prev total WAL file size 86166, number of live WAL files 2.
>
>2018-07-11 21:34:04.708477 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:34:04.708525 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.708519) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:34:04.708534 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1269] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:34:04.708539 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1268 Base level 0, inputs: [11030(83KB)], [11028(16MB)]
>
>2018-07-11 21:34:04.708552 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844708542, "job": 1269, "event": "compaction_started", "files_L0": [11030], "files_L1": [11028], "score": -1, "input_data_size": 17125167}
>2018-07-11 21:34:04.892404 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1269] Generated table #11031: 3188 keys, 15244948 bytes
>2018-07-11 21:34:04.892442 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844892426, "cf_name": "default", "job": 1269, "event": "table_file_creation", "file_number": 11031, "file_size": 15244948, "table_properties": {"data_size": 15127650, "index_size": 67087, "filter_size": 49284, "raw_key_size": 65333, "raw_average_key_size": 20, "raw_value_size": 15034918, "raw_average_value_size": 4716, "num_data_blocks": 1945, "num_entries": 3188, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:34:04.933914 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1269] Compacted 1@0 + 1@1 files to L1 => 15244948 bytes
>2018-07-11 21:34:04.975767 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.975701) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.0 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.3) out(14.5), read-write-amplify(378.6) write-amplify(178.3) OK, records in: 3696, records dropped: 508
>
>2018-07-11 21:34:04.975776 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:34:04.975723) EVENT_LOG_v1 {"time_micros": 1531344844975713, "job": 1269, "event": "compaction_finished", "compaction_time_micros": 225346, "output_level": 1, "num_output_files": 1, "total_output_size": 15244948, "num_input_records": 3696, "num_output_records": 3188, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:34:04.975864 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844975859, "job": 1269, "event": "table_file_deletion", "file_number": 11030}
>2018-07-11 21:34:04.978945 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531344844978940, "job": 1269, "event": "table_file_deletion", "file_number": 11028}
>2018-07-11 21:34:08.501312 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:35:08.515104 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:36:08.553671 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:37:08.575812 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:38:08.596501 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:39:08.618673 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:40:08.649338 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:41:08.679855 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:41:23.806093 7f29149e4700  1 mon.magna102@0(leader).log v21977 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21977)
>2018-07-11 21:41:33.815293 7f29149e4700  1 mon.magna102@0(leader).log v21978 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21978)
>2018-07-11 21:42:03.793725 7f29149e4700  1 mon.magna102@0(leader).log v21979 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21979)
>2018-07-11 21:42:08.702389 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:42:15.770254 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11032. Immutable memtables: 0.
>
>2018-07-11 21:42:15.770298 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1270] Syncing log #11029
>2018-07-11 21:42:15.786540 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.770288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:42:15.786551 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1270] Flushing memtable with next log file: 11032
>
>2018-07-11 21:42:15.786577 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345335786557, "job": 1270, "event": "flush_started", "num_memtables": 1, "num_entries": 1994, "num_deletes": 250, "memory_usage": 3715576}
>2018-07-11 21:42:15.786580 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1270] Level-0 flush table #11033: started
>2018-07-11 21:42:15.837096 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345335837081, "cf_name": "default", "job": 1270, "event": "table_file_creation", "file_number": 11033, "file_size": 1926834, "table_properties": {"data_size": 1910953, "index_size": 8259, "filter_size": 6702, "raw_key_size": 10944, "raw_average_key_size": 21, "raw_value_size": 1897851, "raw_average_value_size": 3656, "num_data_blocks": 259, "num_entries": 519, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:42:15.837106 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1270] Level-0 flush table #11033: 1926834 bytes OK
>2018-07-11 21:42:15.895450 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.853622) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11033 started
>2018-07-11 21:42:15.895458 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.895414) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11033: memtable #1 done
>2018-07-11 21:42:15.895461 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.895429) EVENT_LOG_v1 {"time_micros": 1531345335895423, "job": 1270, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:42:15.895476 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.895441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:42:15.895482 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1270] Try to delete WAL files size 3676480, prev total WAL file size 3676480, number of live WAL files 2.
>
>2018-07-11 21:42:15.896351 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:42:15.896402 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:15.896395) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:42:15.896412 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1271] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:42:15.896418 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1270 Base level 0, inputs: [11033(1881KB)], [11031(14MB)]
>
>2018-07-11 21:42:15.896431 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345335896421, "job": 1271, "event": "compaction_started", "files_L0": [11033], "files_L1": [11031], "score": -1, "input_data_size": 17171782}
>2018-07-11 21:42:16.104498 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1271] Generated table #11034: 3438 keys, 17091822 bytes
>2018-07-11 21:42:16.104542 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345336104515, "cf_name": "default", "job": 1271, "event": "table_file_creation", "file_number": 11034, "file_size": 17091822, "table_properties": {"data_size": 16960591, "index_size": 75210, "filter_size": 55094, "raw_key_size": 70326, "raw_average_key_size": 20, "raw_value_size": 16858680, "raw_average_value_size": 4903, "num_data_blocks": 2192, "num_entries": 3438, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:42:16.137671 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1271] Compacted 1@0 + 1@1 files to L1 => 17091822 bytes
>2018-07-11 21:42:16.179398 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:16.179331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.2 rd, 70.9 wr, level 1, files in(1, 1) out(1) MB in(1.8, 14.5) out(16.3), read-write-amplify(17.8) write-amplify(8.9) OK, records in: 3707, records dropped: 269
>
>2018-07-11 21:42:16.179407 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:16.179352) EVENT_LOG_v1 {"time_micros": 1531345336179343, "job": 1271, "event": "compaction_finished", "compaction_time_micros": 241225, "output_level": 1, "num_output_files": 1, "total_output_size": 17091822, "num_input_records": 3707, "num_output_records": 3438, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:42:16.179839 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345336179835, "job": 1271, "event": "table_file_deletion", "file_number": 11033}
>2018-07-11 21:42:16.182881 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345336182877, "job": 1271, "event": "table_file_deletion", "file_number": 11031}
>2018-07-11 21:42:16.182996 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:42:16.905495 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11035. Immutable memtables: 0.
>
>2018-07-11 21:42:16.905534 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1272] Syncing log #11032
>2018-07-11 21:42:16.922100 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:16.905525) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:42:16.922112 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1272] Flushing memtable with next log file: 11035
>
>2018-07-11 21:42:16.922137 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345336922117, "job": 1272, "event": "flush_started", "num_memtables": 1, "num_entries": 263, "num_deletes": 251, "memory_usage": 34704}
>2018-07-11 21:42:16.922140 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1272] Level-0 flush table #11036: started
>2018-07-11 21:42:16.964271 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345336964258, "cf_name": "default", "job": 1272, "event": "table_file_creation", "file_number": 11036, "file_size": 28567, "table_properties": {"data_size": 26810, "index_size": 115, "filter_size": 724, "raw_key_size": 5238, "raw_average_key_size": 20, "raw_value_size": 23267, "raw_average_value_size": 89, "num_data_blocks": 4, "num_entries": 260, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:42:16.964281 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1272] Level-0 flush table #11036: 28567 bytes OK
>2018-07-11 21:42:17.014304 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:16.980823) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11036 started
>2018-07-11 21:42:17.014312 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.014271) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11036: memtable #1 done
>2018-07-11 21:42:17.014315 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.014285) EVENT_LOG_v1 {"time_micros": 1531345337014279, "job": 1272, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:42:17.014330 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.014296) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:42:17.014335 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1272] Try to delete WAL files size 27090, prev total WAL file size 27090, number of live WAL files 2.
>
>2018-07-11 21:42:17.014433 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:42:17.014480 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.014474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:42:17.014489 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1273] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:42:17.014495 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1272 Base level 0, inputs: [11036(27KB)], [11034(16MB)]
>
>2018-07-11 21:42:17.014508 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345337014498, "job": 1273, "event": "compaction_started", "files_L0": [11036], "files_L1": [11034], "score": -1, "input_data_size": 17120389}
>2018-07-11 21:42:17.198333 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1273] Generated table #11037: 3190 keys, 15246328 bytes
>2018-07-11 21:42:17.198382 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345337198353, "cf_name": "default", "job": 1273, "event": "table_file_creation", "file_number": 11037, "file_size": 15246328, "table_properties": {"data_size": 15129016, "index_size": 67090, "filter_size": 49295, "raw_key_size": 65368, "raw_average_key_size": 20, "raw_value_size": 15036255, "raw_average_value_size": 4713, "num_data_blocks": 1945, "num_entries": 3190, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:42:17.231518 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1273] Compacted 1@0 + 1@1 files to L1 => 15246328 bytes
>2018-07-11 21:42:17.265046 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.264978) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.9 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.0, 16.3) out(14.5), read-write-amplify(1133.0) write-amplify(533.7) OK, records in: 3698, records dropped: 508
>
>2018-07-11 21:42:17.265055 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:42:17.265000) EVENT_LOG_v1 {"time_micros": 1531345337264991, "job": 1273, "event": "compaction_finished", "compaction_time_micros": 216993, "output_level": 1, "num_output_files": 1, "total_output_size": 15246328, "num_input_records": 3698, "num_output_records": 3190, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:42:17.265150 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345337265146, "job": 1273, "event": "table_file_deletion", "file_number": 11036}
>2018-07-11 21:42:17.268298 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345337268295, "job": 1273, "event": "table_file_deletion", "file_number": 11034}
>2018-07-11 21:43:08.736809 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:44:08.758822 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:45:08.789197 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:46:08.802763 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:47:08.840304 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:47:53.961509 7f29149e4700  1 mon.magna102@0(leader).log v21980 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21980)
>2018-07-11 21:48:08.862492 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:49:08.893072 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:50:08.915297 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:50:35.059434 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11038. Immutable memtables: 0.
>
>2018-07-11 21:50:35.059478 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1274] Syncing log #11035
>2018-07-11 21:50:35.075979 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.059468) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:50:35.075990 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1274] Flushing memtable with next log file: 11038
>
>2018-07-11 21:50:35.076018 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835076009, "job": 1274, "event": "flush_started", "num_memtables": 1, "num_entries": 2007, "num_deletes": 251, "memory_usage": 3744136}
>2018-07-11 21:50:35.076021 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1274] Level-0 flush table #11039: started
>2018-07-11 21:50:35.143215 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835143200, "cf_name": "default", "job": 1274, "event": "table_file_creation", "file_number": 11039, "file_size": 3572993, "table_properties": {"data_size": 3543307, "index_size": 16687, "filter_size": 12074, "raw_key_size": 15766, "raw_average_key_size": 20, "raw_value_size": 3520775, "raw_average_value_size": 4626, "num_data_blocks": 502, "num_entries": 761, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:50:35.143225 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1274] Level-0 flush table #11039: 3572993 bytes OK
>2018-07-11 21:50:35.193204 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.159723) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11039 started
>2018-07-11 21:50:35.193212 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.193158) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11039: memtable #1 done
>2018-07-11 21:50:35.193214 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.193172) EVENT_LOG_v1 {"time_micros": 1531345835193166, "job": 1274, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:50:35.193229 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.193193) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:50:35.193235 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1274] Try to delete WAL files size 3704348, prev total WAL file size 3704348, number of live WAL files 2.
>
>2018-07-11 21:50:35.194124 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:50:35.194331 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 21:50:35.194337 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 107862.1 total, 999.5 interval
>Cumulative writes: 134K writes, 628K keys, 134K commit groups, 1.0 writes per commit group, ingest: 1.02 GB, 0.01 MB/s
>Cumulative WAL: 134K writes, 134K syncs, 1.00 writes per sync, written: 1.02 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1016 writes, 4555 keys, 1016 commit groups, 1.0 writes per commit group, ingest: 7.15 MB, 0.01 MB/s
>Interval WAL: 1016 writes, 1016 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.41 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.5        50       638    0.078       0      0
>  L1      1/1   14.54 MB   0.0     10.1     0.8      9.2       9.2     -0.0       0.0  10.9     72.5     66.4       142       635    0.224   2283K   276K
> Sum      2/2   17.95 MB   0.0     10.1     0.8      9.2      10.1      0.9       0.0  11.8     53.7     53.7       192      1273    0.151   2283K   276K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  12.5     55.6     56.9         1         8    0.147     14K   1553
>Uptime(secs): 107862.1 total, 107862.1 interval
>Flush(GB): cumulative 0.851, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.06 GB write, 0.10 MB/s write, 10.05 GB read, 0.10 MB/s read, 191.7 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.41 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.5        50       638    0.078       0      0
>  L1      1/1   14.54 MB   0.0     10.1     0.8      9.2       9.2     -0.0       0.0  10.9     72.5     66.4       142       635    0.224   2283K   276K
> Sum      2/2   17.95 MB   0.0     10.1     0.8      9.2      10.1      0.9       0.0  11.8     53.7     53.7       192      1273    0.151   2283K   276K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 107862.1 total, 107862.1 interval
>Flush(GB): cumulative 0.851, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.06 GB write, 0.10 MB/s write, 10.05 GB read, 0.10 MB/s read, 191.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 21:50:35.194359 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.194352) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:50:35.194364 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1275] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:50:35.194369 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1274 Base level 0, inputs: [11039(3489KB)], [11037(14MB)]
>
>2018-07-11 21:50:35.194382 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835194373, "job": 1275, "event": "compaction_started", "files_L0": [11039], "files_L1": [11037], "score": -1, "input_data_size": 18819321}
>2018-07-11 21:50:35.393913 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1275] Generated table #11040: 3441 keys, 16939383 bytes
>2018-07-11 21:50:35.393956 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835393930, "cf_name": "default", "job": 1275, "event": "table_file_creation", "file_number": 11040, "file_size": 16939383, "table_properties": {"data_size": 16807781, "index_size": 75845, "filter_size": 54830, "raw_key_size": 70887, "raw_average_key_size": 20, "raw_value_size": 16705232, "raw_average_value_size": 4854, "num_data_blocks": 2196, "num_entries": 3441, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:50:35.427110 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1275] Compacted 1@0 + 1@1 files to L1 => 16939383 bytes
>2018-07-11 21:50:35.460634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.460565) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.9 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(3.4, 14.5) out(16.2), read-write-amplify(10.0) write-amplify(4.7) OK, records in: 3951, records dropped: 510
>
>2018-07-11 21:50:35.460643 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.460587) EVENT_LOG_v1 {"time_micros": 1531345835460577, "job": 1275, "event": "compaction_finished", "compaction_time_micros": 232712, "output_level": 1, "num_output_files": 1, "total_output_size": 16939383, "num_input_records": 3951, "num_output_records": 3441, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:50:35.461423 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835461419, "job": 1275, "event": "table_file_deletion", "file_number": 11039}
>2018-07-11 21:50:35.464434 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345835464431, "job": 1275, "event": "table_file_deletion", "file_number": 11037}
>2018-07-11 21:50:35.996024 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11041. Immutable memtables: 0.
>
>2018-07-11 21:50:35.996089 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1276] Syncing log #11038
>2018-07-11 21:50:36.012670 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:35.996078) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:50:36.012701 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1276] Flushing memtable with next log file: 11041
>
>2018-07-11 21:50:36.012715 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836012708, "job": 1276, "event": "flush_started", "num_memtables": 1, "num_entries": 255, "num_deletes": 250, "memory_usage": 20240}
>2018-07-11 21:50:36.012718 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1276] Level-0 flush table #11042: started
>2018-07-11 21:50:36.054747 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836054734, "cf_name": "default", "job": 1276, "event": "table_file_creation", "file_number": 11042, "file_size": 14277, "table_properties": {"data_size": 12637, "index_size": 53, "filter_size": 673, "raw_key_size": 5626, "raw_average_key_size": 22, "raw_value_size": 9205, "raw_average_value_size": 36, "num_data_blocks": 2, "num_entries": 255, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:50:36.054757 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1276] Level-0 flush table #11042: 14277 bytes OK
>2018-07-11 21:50:36.104788 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.071304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11042 started
>2018-07-11 21:50:36.104796 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.104754) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11042: memtable #1 done
>2018-07-11 21:50:36.104798 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.104769) EVENT_LOG_v1 {"time_micros": 1531345836104763, "job": 1276, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:50:36.104813 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.104780) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:50:36.104818 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1276] Try to delete WAL files size 13331, prev total WAL file size 13331, number of live WAL files 2.
>
>2018-07-11 21:50:36.104917 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:50:36.104964 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.104957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:50:36.104973 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1277] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:50:36.104979 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1276 Base level 0, inputs: [11042(13KB)], [11040(16MB)]
>
>2018-07-11 21:50:36.104992 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836104982, "job": 1277, "event": "compaction_started", "files_L0": [11042], "files_L1": [11040], "score": -1, "input_data_size": 16953660}
>2018-07-11 21:50:36.288809 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1277] Generated table #11043: 3192 keys, 15254145 bytes
>2018-07-11 21:50:36.288859 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836288831, "cf_name": "default", "job": 1277, "event": "table_file_creation", "file_number": 11043, "file_size": 15254145, "table_properties": {"data_size": 15136741, "index_size": 67152, "filter_size": 49325, "raw_key_size": 65407, "raw_average_key_size": 20, "raw_value_size": 15043905, "raw_average_value_size": 4713, "num_data_blocks": 1947, "num_entries": 3192, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:50:36.321994 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1277] Compacted 1@0 + 1@1 files to L1 => 15254145 bytes
>2018-07-11 21:50:36.355528 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.355460) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.1 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.0, 16.2) out(14.5), read-write-amplify(2255.9) write-amplify(1068.4) OK, records in: 3696, records dropped: 504
>
>2018-07-11 21:50:36.355537 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:50:36.355482) EVENT_LOG_v1 {"time_micros": 1531345836355472, "job": 1277, "event": "compaction_finished", "compaction_time_micros": 216988, "output_level": 1, "num_output_files": 1, "total_output_size": 15254145, "num_input_records": 3696, "num_output_records": 3192, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:50:36.355631 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836355627, "job": 1277, "event": "table_file_deletion", "file_number": 11042}
>2018-07-11 21:50:36.358779 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531345836358776, "job": 1277, "event": "table_file_deletion", "file_number": 11040}
>2018-07-11 21:50:36.358896 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:51:08.949949 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:51:18.042443 7f29149e4700  1 mon.magna102@0(leader).log v21981 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21981)
>2018-07-11 21:51:36.920273 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:51:37.070602 7f29149e4700  1 mon.magna102@0(leader).log v21982 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21982)
>2018-07-11 21:51:46.904309 7f29149e4700  1 mon.magna102@0(leader).log v21983 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21983)
>2018-07-11 21:51:56.905176 7f29149e4700  1 mon.magna102@0(leader).log v21984 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 21984)
>2018-07-11 21:52:08.972754 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:52:46.951880 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 21:52:47.126916 7f29149e4700  1 mon.magna102@0(leader).log v21985 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21985)
>2018-07-11 21:52:50.953526 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 21:52:51.127367 7f29149e4700  1 mon.magna102@0(leader).log v21986 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21986)
>2018-07-11 21:53:08.995088 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:53:10.962420 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:53:11.112416 7f29149e4700  1 mon.magna102@0(leader).log v21987 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21987)
>2018-07-11 21:54:09.025649 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:54:20.993937 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 21:54:21.160288 7f29149e4700  1 mon.magna102@0(leader).log v21988 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21988)
>2018-07-11 21:54:30.998405 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 21:54:31.169460 7f29149e4700  1 mon.magna102@0(leader).log v21989 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21989)
>2018-07-11 21:54:51.007336 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:54:51.162660 7f29149e4700  1 mon.magna102@0(leader).log v21990 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21990)
>2018-07-11 21:55:09.048244 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:56:01.038994 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 21:56:01.193735 7f29149e4700  1 mon.magna102@0(leader).log v21991 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21991)
>2018-07-11 21:56:09.077421 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:56:11.043437 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 21:56:11.212471 7f29149e4700  1 mon.magna102@0(leader).log v21992 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21992)
>2018-07-11 21:56:27.050387 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:56:27.222025 7f29149e4700  1 mon.magna102@0(leader).log v21993 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21993)
>2018-07-11 21:57:09.099675 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:57:37.081817 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 21:57:37.236515 7f29149e4700  1 mon.magna102@0(leader).log v21994 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21994)
>2018-07-11 21:57:47.086316 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 21:57:47.237331 7f29149e4700  1 mon.magna102@0(leader).log v21995 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21995)
>2018-07-11 21:58:07.095155 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:58:07.247133 7f29149e4700  1 mon.magna102@0(leader).log v21996 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21996)
>2018-07-11 21:58:09.130422 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 21:58:21.256910 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11044. Immutable memtables: 0.
>
>2018-07-11 21:58:21.256955 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1278] Syncing log #11041
>2018-07-11 21:58:21.273488 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.256946) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:58:21.273499 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1278] Flushing memtable with next log file: 11044
>
>2018-07-11 21:58:21.273513 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301273506, "job": 1278, "event": "flush_started", "num_memtables": 1, "num_entries": 2015, "num_deletes": 251, "memory_usage": 3799672}
>2018-07-11 21:58:21.273516 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1278] Level-0 flush table #11045: started
>2018-07-11 21:58:21.349049 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301349034, "cf_name": "default", "job": 1278, "event": "table_file_creation", "file_number": 11045, "file_size": 3610344, "table_properties": {"data_size": 3580586, "index_size": 16661, "filter_size": 12172, "raw_key_size": 16009, "raw_average_key_size": 20, "raw_value_size": 3557886, "raw_average_value_size": 4596, "num_data_blocks": 501, "num_entries": 774, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 21:58:21.349073 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1278] Level-0 flush table #11045: 3610344 bytes OK
>2018-07-11 21:58:21.399034 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.365545) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11045 started
>2018-07-11 21:58:21.399043 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.398995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11045: memtable #1 done
>2018-07-11 21:58:21.399046 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.399010) EVENT_LOG_v1 {"time_micros": 1531346301399003, "job": 1278, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:58:21.399051 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.399023) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:58:21.399057 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1278] Try to delete WAL files size 3761043, prev total WAL file size 3761043, number of live WAL files 2.
>
>2018-07-11 21:58:21.399947 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:58:21.400001 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.399994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 21:58:21.400010 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1279] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:58:21.400016 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1278 Base level 0, inputs: [11045(3525KB)], [11043(14MB)]
>
>2018-07-11 21:58:21.400030 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301400019, "job": 1279, "event": "compaction_started", "files_L0": [11045], "files_L1": [11043], "score": -1, "input_data_size": 18864489}
>2018-07-11 21:58:21.600984 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1279] Generated table #11046: 3456 keys, 16984177 bytes
>2018-07-11 21:58:21.601028 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301601000, "cf_name": "default", "job": 1279, "event": "table_file_creation", "file_number": 11046, "file_size": 16984177, "table_properties": {"data_size": 16852485, "index_size": 75821, "filter_size": 54944, "raw_key_size": 71169, "raw_average_key_size": 20, "raw_value_size": 16749745, "raw_average_value_size": 4846, "num_data_blocks": 2195, "num_entries": 3456, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:58:21.634193 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1279] Compacted 1@0 + 1@1 files to L1 => 16984177 bytes
>2018-07-11 21:58:21.667704 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.667633) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.6 rd, 72.5 wr, level 1, files in(1, 1) out(1) MB in(3.4, 14.5) out(16.2), read-write-amplify(9.9) write-amplify(4.7) OK, records in: 3966, records dropped: 510
>
>2018-07-11 21:58:21.667713 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:21.667654) EVENT_LOG_v1 {"time_micros": 1531346301667644, "job": 1279, "event": "compaction_finished", "compaction_time_micros": 234142, "output_level": 1, "num_output_files": 1, "total_output_size": 16984177, "num_input_records": 3966, "num_output_records": 3456, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:58:21.668456 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301668452, "job": 1279, "event": "table_file_deletion", "file_number": 11045}
>2018-07-11 21:58:21.671435 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346301671432, "job": 1279, "event": "table_file_deletion", "file_number": 11043}
>2018-07-11 21:58:56.237452 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11047. Immutable memtables: 0.
>
>2018-07-11 21:58:56.237494 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1280] Syncing log #11044
>2018-07-11 21:58:56.254015 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.237485) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 21:58:56.254026 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1280] Flushing memtable with next log file: 11047
>
>2018-07-11 21:58:56.254039 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336254032, "job": 1280, "event": "flush_started", "num_memtables": 1, "num_entries": 374, "num_deletes": 250, "memory_usage": 282528}
>2018-07-11 21:58:56.254042 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1280] Level-0 flush table #11048: started
>2018-07-11 21:58:56.296222 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336296209, "cf_name": "default", "job": 1280, "event": "table_file_creation", "file_number": 11048, "file_size": 187602, "table_properties": {"data_size": 184639, "index_size": 827, "filter_size": 1217, "raw_key_size": 6150, "raw_average_key_size": 21, "raw_value_size": 180267, "raw_average_value_size": 643, "num_data_blocks": 26, "num_entries": 280, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 21:58:56.296231 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1280] Level-0 flush table #11048: 187602 bytes OK
>2018-07-11 21:58:56.346237 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.312764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11048 started
>2018-07-11 21:58:56.346246 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.346203) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11048: memtable #1 done
>2018-07-11 21:58:56.346248 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.346217) EVENT_LOG_v1 {"time_micros": 1531346336346211, "job": 1280, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 21:58:56.346250 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.346229) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 21:58:56.346256 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1280] Try to delete WAL files size 273500, prev total WAL file size 273500, number of live WAL files 2.
>
>2018-07-11 21:58:56.346389 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:58:56.346439 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.346433) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 21:58:56.346448 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1281] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 21:58:56.346454 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1280 Base level 0, inputs: [11048(183KB)], [11046(16MB)]
>
>2018-07-11 21:58:56.346466 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336346458, "job": 1281, "event": "compaction_started", "files_L0": [11048], "files_L1": [11046], "score": -1, "input_data_size": 17171779}
>2018-07-11 21:58:56.530275 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1281] Generated table #11049: 3241 keys, 15530122 bytes
>2018-07-11 21:58:56.530312 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336530296, "cf_name": "default", "job": 1281, "event": "table_file_creation", "file_number": 11049, "file_size": 15530122, "table_properties": {"data_size": 15410716, "index_size": 68286, "filter_size": 50193, "raw_key_size": 66403, "raw_average_key_size": 20, "raw_value_size": 15316397, "raw_average_value_size": 4725, "num_data_blocks": 1980, "num_entries": 3241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 21:58:56.563466 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1281] Compacted 1@0 + 1@1 files to L1 => 15530122 bytes
>2018-07-11 21:58:56.596996 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.596926) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.1 rd, 71.6 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.2) out(14.8), read-write-amplify(174.3) write-amplify(82.8) OK, records in: 3736, records dropped: 495
>
>2018-07-11 21:58:56.597005 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-21:58:56.596948) EVENT_LOG_v1 {"time_micros": 1531346336596938, "job": 1281, "event": "compaction_finished", "compaction_time_micros": 216982, "output_level": 1, "num_output_files": 1, "total_output_size": 15530122, "num_input_records": 3736, "num_output_records": 3241, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 21:58:56.597106 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336597102, "job": 1281, "event": "table_file_deletion", "file_number": 11048}
>2018-07-11 21:58:56.600151 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346336600148, "job": 1281, "event": "table_file_deletion", "file_number": 11046}
>2018-07-11 21:58:56.600206 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 21:59:09.156520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 21:59:17.127131 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 21:59:17.275187 7f29149e4700  1 mon.magna102@0(leader).log v21997 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21997)
>2018-07-11 21:59:27.131533 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 21:59:27.284468 7f29149e4700  1 mon.magna102@0(leader).log v21998 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21998)
>2018-07-11 21:59:41.138019 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 21:59:41.294067 7f29149e4700  1 mon.magna102@0(leader).log v21999 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 21999)
>2018-07-11 21:59:47.140530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 21:59:47.294634 7f29149e4700  1 mon.magna102@0(leader).log v22000 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22000)
>2018-07-11 22:00:09.187883 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:00:57.172442 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:00:57.326099 7f29149e4700  1 mon.magna102@0(leader).log v22001 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22001)
>2018-07-11 22:01:07.177008 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:01:07.327224 7f29149e4700  1 mon.magna102@0(leader).log v22002 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22002)
>2018-07-11 22:01:09.210597 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:01:21.183413 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 22:01:21.338082 7f29149e4700  1 mon.magna102@0(leader).log v22003 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22003)
>2018-07-11 22:02:09.232772 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:02:37.217450 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:02:37.369492 7f29149e4700  1 mon.magna102@0(leader).log v22004 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22004)
>2018-07-11 22:02:41.219304 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:02:41.361598 7f29149e4700  1 mon.magna102@0(leader).log v22005 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22005)
>2018-07-11 22:03:01.228219 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 22:03:01.388110 7f29149e4700  1 mon.magna102@0(leader).log v22006 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22006)
>2018-07-11 22:03:09.263431 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:04:09.293657 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:04:11.259756 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:04:11.410830 7f29149e4700  1 mon.magna102@0(leader).log v22007 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22007)
>2018-07-11 22:04:21.264268 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:04:21.411651 7f29149e4700  1 mon.magna102@0(leader).log v22008 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22008)
>2018-07-11 22:04:37.271234 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 22:04:37.421229 7f29149e4700  1 mon.magna102@0(leader).log v22009 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22009)
>2018-07-11 22:05:09.324337 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:05:57.307256 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:05:57.486252 7f29149e4700  1 mon.magna102@0(leader).log v22010 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22010)
>2018-07-11 22:06:01.309219 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:06:01.478346 7f29149e4700  1 mon.magna102@0(leader).log v22011 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22011)
>2018-07-11 22:06:09.345314 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:06:09.445960 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11050. Immutable memtables: 0.
>
>2018-07-11 22:06:09.446005 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1282] Syncing log #11047
>2018-07-11 22:06:09.463744 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.445995) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:06:09.463756 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1282] Flushing memtable with next log file: 11050
>
>2018-07-11 22:06:09.463782 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769463774, "job": 1282, "event": "flush_started", "num_memtables": 1, "num_entries": 1895, "num_deletes": 251, "memory_usage": 3573744}
>2018-07-11 22:06:09.463785 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1282] Level-0 flush table #11051: started
>2018-07-11 22:06:09.530926 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769530912, "cf_name": "default", "job": 1282, "event": "table_file_creation", "file_number": 11051, "file_size": 3392080, "table_properties": {"data_size": 3364238, "index_size": 15485, "filter_size": 11433, "raw_key_size": 15276, "raw_average_key_size": 20, "raw_value_size": 3342864, "raw_average_value_size": 4523, "num_data_blocks": 466, "num_entries": 739, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:06:09.530950 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1282] Level-0 flush table #11051: 3392080 bytes OK
>2018-07-11 22:06:09.580977 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.547436) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11051 started
>2018-07-11 22:06:09.580986 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.580903) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11051: memtable #1 done
>2018-07-11 22:06:09.580989 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.580918) EVENT_LOG_v1 {"time_micros": 1531346769580912, "job": 1282, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:06:09.580991 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.580931) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:06:09.581013 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1282] Try to delete WAL files size 3537370, prev total WAL file size 3537370, number of live WAL files 2.
>
>2018-07-11 22:06:09.581873 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:06:09.582068 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 22:06:09.582075 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 108796.4 total, 934.4 interval
>Cumulative writes: 135K writes, 632K keys, 135K commit groups, 1.0 writes per commit group, ingest: 1.03 GB, 0.01 MB/s
>Cumulative WAL: 135K writes, 135K syncs, 1.00 writes per sync, written: 1.03 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4539 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.23 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.23 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        50       642    0.078       0      0
>  L1      1/1   14.81 MB   0.0     10.1     0.9      9.3       9.3     -0.0       0.0  10.8     72.5     66.4       143       639    0.224   2298K   278K
> Sum      2/2   18.05 MB   0.0     10.1     0.9      9.3      10.1      0.9       0.0  11.8     53.7     53.7       193      1281    0.151   2298K   278K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  10.0     57.4     57.4         1         8    0.149     15K   2019
>Uptime(secs): 108796.4 total, 108796.4 interval
>Flush(GB): cumulative 0.858, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.12 GB write, 0.10 MB/s write, 10.12 GB read, 0.10 MB/s read, 192.9 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.23 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        50       642    0.078       0      0
>  L1      1/1   14.81 MB   0.0     10.1     0.9      9.3       9.3     -0.0       0.0  10.8     72.5     66.4       143       639    0.224   2298K   278K
> Sum      2/2   18.05 MB   0.0     10.1     0.9      9.3      10.1      0.9       0.0  11.8     53.7     53.7       193      1281    0.151   2298K   278K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 108796.4 total, 108796.4 interval
>Flush(GB): cumulative 0.858, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.12 GB write, 0.10 MB/s write, 10.12 GB read, 0.10 MB/s read, 192.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 22:06:09.582097 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.582090) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:06:09.582102 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1283] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:06:09.582107 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1282 Base level 0, inputs: [11051(3312KB)], [11049(14MB)]
>
>2018-07-11 22:06:09.582120 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769582111, "job": 1283, "event": "compaction_started", "files_L0": [11051], "files_L1": [11049], "score": -1, "input_data_size": 18922202}
>2018-07-11 22:06:09.773321 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1283] Generated table #11052: 3470 keys, 17043922 bytes
>2018-07-11 22:06:09.773366 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769773350, "cf_name": "default", "job": 1283, "event": "table_file_creation", "file_number": 11052, "file_size": 17043922, "table_properties": {"data_size": 16912072, "index_size": 75833, "filter_size": 55090, "raw_key_size": 71432, "raw_average_key_size": 20, "raw_value_size": 16809131, "raw_average_value_size": 4844, "num_data_blocks": 2195, "num_entries": 3470, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:06:09.806494 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1283] Compacted 1@0 + 1@1 files to L1 => 17043922 bytes
>2018-07-11 22:06:09.840006 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.839937) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 84.3 rd, 76.0 wr, level 1, files in(1, 1) out(1) MB in(3.2, 14.8) out(16.3), read-write-amplify(10.6) write-amplify(5.0) OK, records in: 3980, records dropped: 510
>
>2018-07-11 22:06:09.840015 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:06:09.839959) EVENT_LOG_v1 {"time_micros": 1531346769839949, "job": 1283, "event": "compaction_finished", "compaction_time_micros": 224360, "output_level": 1, "num_output_files": 1, "total_output_size": 17043922, "num_input_records": 3980, "num_output_records": 3470, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:06:09.840720 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769840716, "job": 1283, "event": "table_file_deletion", "file_number": 11051}
>2018-07-11 22:06:09.843815 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346769843812, "job": 1283, "event": "table_file_deletion", "file_number": 11049}
>2018-07-11 22:06:17.316274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
>2018-07-11 22:06:17.466212 7f29149e4700  1 mon.magna102@0(leader).log v22012 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22012)
>2018-07-11 22:06:26.349785 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,6 (REQUEST_SLOW)
>2018-07-11 22:06:26.458535 7f29149e4700  1 mon.magna102@0(leader).log v22013 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22013)
>2018-07-11 22:06:51.331705 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:06:51.485636 7f29149e4700  1 mon.magna102@0(leader).log v22014 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22014)
>2018-07-11 22:06:57.085955 7f29149e4700  1 mon.magna102@0(leader).log v22015 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22015)
>2018-07-11 22:07:01.336261 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:07:01.496217 7f29149e4700  1 mon.magna102@0(leader).log v22016 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22016)
>2018-07-11 22:07:09.379905 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 22:07:16.455831 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11053. Immutable memtables: 0.
>
>2018-07-11 22:07:16.455875 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1284] Syncing log #11050
>2018-07-11 22:07:16.472368 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.455865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:07:16.472379 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1284] Flushing memtable with next log file: 11053
>
>2018-07-11 22:07:16.472405 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836472397, "job": 1284, "event": "flush_started", "num_memtables": 1, "num_entries": 526, "num_deletes": 250, "memory_usage": 606768}
>2018-07-11 22:07:16.472408 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1284] Level-0 flush table #11054: started
>2018-07-11 22:07:16.514551 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836514538, "cf_name": "default", "job": 1284, "event": "table_file_creation", "file_number": 11054, "file_size": 390308, "table_properties": {"data_size": 385900, "index_size": 1634, "filter_size": 1855, "raw_key_size": 6826, "raw_average_key_size": 21, "raw_value_size": 380463, "raw_average_value_size": 1215, "num_data_blocks": 51, "num_entries": 313, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:07:16.514561 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1284] Level-0 flush table #11054: 390308 bytes OK
>2018-07-11 22:07:16.564597 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.531094) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11054 started
>2018-07-11 22:07:16.564605 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.564562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11054: memtable #1 done
>2018-07-11 22:07:16.564608 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.564576) EVENT_LOG_v1 {"time_micros": 1531346836564570, "job": 1284, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:07:16.564610 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.564588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:07:16.564629 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1284] Try to delete WAL files size 594897, prev total WAL file size 594897, number of live WAL files 2.
>
>2018-07-11 22:07:16.564910 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:07:16.564960 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.564953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:07:16.564970 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1285] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:07:16.564977 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1284 Base level 0, inputs: [11054(381KB)], [11052(16MB)]
>
>2018-07-11 22:07:16.564990 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836564980, "job": 1285, "event": "compaction_started", "files_L0": [11054], "files_L1": [11052], "score": -1, "input_data_size": 17434230}
>2018-07-11 22:07:16.756989 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1285] Generated table #11055: 3302 keys, 15900388 bytes
>2018-07-11 22:07:16.757033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836757006, "cf_name": "default", "job": 1285, "event": "table_file_creation", "file_number": 11055, "file_size": 15900388, "table_properties": {"data_size": 15778404, "index_size": 69724, "filter_size": 51333, "raw_key_size": 67643, "raw_average_key_size": 20, "raw_value_size": 15682209, "raw_average_value_size": 4749, "num_data_blocks": 2022, "num_entries": 3302, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:07:16.790148 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1285] Compacted 1@0 + 1@1 files to L1 => 15900388 bytes
>2018-07-11 22:07:16.823686 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.823618) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.4 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.3) out(15.2), read-write-amplify(85.4) write-amplify(40.7) OK, records in: 3783, records dropped: 481
>
>2018-07-11 22:07:16.823695 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:07:16.823640) EVENT_LOG_v1 {"time_micros": 1531346836823630, "job": 1285, "event": "compaction_finished", "compaction_time_micros": 225144, "output_level": 1, "num_output_files": 1, "total_output_size": 15900388, "num_input_records": 3783, "num_output_records": 3302, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:07:16.823849 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836823844, "job": 1285, "event": "table_file_deletion", "file_number": 11054}
>2018-07-11 22:07:16.826984 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531346836826981, "job": 1285, "event": "table_file_deletion", "file_number": 11052}
>2018-07-11 22:07:16.827100 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:08:09.403967 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:08:53.548990 7f29149e4700  1 mon.magna102@0(leader).log v22017 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22017)
>2018-07-11 22:09:05.390913 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:09:05.541674 7f29149e4700  1 mon.magna102@0(leader).log v22018 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22018)
>2018-07-11 22:09:09.426331 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:09:11.426526 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:09:11.543516 7f29149e4700  1 mon.magna102@0(leader).log v22019 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22019)
>2018-07-11 22:10:09.455624 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:10:25.427056 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 5,6)
>2018-07-11 22:10:25.599942 7f29149e4700  1 mon.magna102@0(leader).log v22020 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22020)
>2018-07-11 22:10:26.550575 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11056. Immutable memtables: 0.
>
>2018-07-11 22:10:26.550619 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1286] Syncing log #11053
>2018-07-11 22:10:26.566843 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.550609) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:10:26.566854 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1286] Flushing memtable with next log file: 11056
>
>2018-07-11 22:10:26.566868 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026566860, "job": 1286, "event": "flush_started", "num_memtables": 1, "num_entries": 1202, "num_deletes": 500, "memory_usage": 1544184}
>2018-07-11 22:10:26.566871 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1286] Level-0 flush table #11057: started
>2018-07-11 22:10:26.617388 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026617374, "cf_name": "default", "job": 1286, "event": "table_file_creation", "file_number": 11057, "file_size": 1462079, "table_properties": {"data_size": 1448633, "index_size": 6622, "filter_size": 5904, "raw_key_size": 14710, "raw_average_key_size": 20, "raw_value_size": 1434230, "raw_average_value_size": 2017, "num_data_blocks": 202, "num_entries": 711, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-11 22:10:26.617398 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1286] Level-0 flush table #11057: 1462079 bytes OK
>2018-07-11 22:10:26.667401 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.633940) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11057 started
>2018-07-11 22:10:26.667409 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.667365) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11057: memtable #1 done
>2018-07-11 22:10:26.667412 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.667379) EVENT_LOG_v1 {"time_micros": 1531347026667373, "job": 1286, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:10:26.667414 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.667391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:10:26.667420 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1286] Try to delete WAL files size 1517614, prev total WAL file size 1517614, number of live WAL files 2.
>
>2018-07-11 22:10:26.667855 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:10:26.667909 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.667902) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-11 22:10:26.667918 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1287] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:10:26.667924 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1286 Base level 0, inputs: [11057(1427KB)], [11055(15MB)]
>
>2018-07-11 22:10:26.667937 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026667927, "job": 1287, "event": "compaction_started", "files_L0": [11057], "files_L1": [11055], "score": -1, "input_data_size": 17362467}
>2018-07-11 22:10:26.851454 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1287] Generated table #11058: 3005 keys, 15102870 bytes
>2018-07-11 22:10:26.851503 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026851487, "cf_name": "default", "job": 1287, "event": "table_file_creation", "file_number": 11058, "file_size": 15102870, "table_properties": {"data_size": 14986090, "index_size": 67152, "filter_size": 48701, "raw_key_size": 61647, "raw_average_key_size": 20, "raw_value_size": 14895960, "raw_average_value_size": 4957, "num_data_blocks": 1959, "num_entries": 3005, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:10:26.884607 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1287] Compacted 1@0 + 1@1 files to L1 => 15102870 bytes
>2018-07-11 22:10:26.918141 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.918070) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.1 rd, 69.7 wr, level 1, files in(1, 1) out(1) MB in(1.4, 15.2) out(14.4), read-write-amplify(22.2) write-amplify(10.3) OK, records in: 4013, records dropped: 1008
>
>2018-07-11 22:10:26.918149 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:10:26.918092) EVENT_LOG_v1 {"time_micros": 1531347026918082, "job": 1287, "event": "compaction_finished", "compaction_time_micros": 216656, "output_level": 1, "num_output_files": 1, "total_output_size": 15102870, "num_input_records": 4013, "num_output_records": 3005, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:10:26.918526 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026918522, "job": 1287, "event": "table_file_deletion", "file_number": 11057}
>2018-07-11 22:10:26.921655 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347026921652, "job": 1287, "event": "table_file_deletion", "file_number": 11055}
>2018-07-11 22:10:26.921709 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:10:41.434514 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:10:41.595032 7f29149e4700  1 mon.magna102@0(leader).log v22021 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22021)
>2018-07-11 22:10:46.509194 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:10:46.603801 7f29149e4700  1 mon.magna102@0(leader).log v22022 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22022)
>2018-07-11 22:11:09.488626 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:12:05.472092 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:12:05.627224 7f29149e4700  1 mon.magna102@0(leader).log v22023 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22023)
>2018-07-11 22:12:09.473604 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:12:09.510664 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:12:09.627690 7f29149e4700  1 mon.magna102@0(leader).log v22024 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22024)
>2018-07-11 22:12:25.480915 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:12:25.646794 7f29149e4700  1 mon.magna102@0(leader).log v22025 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22025)
>2018-07-11 22:13:09.532915 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:13:45.516378 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:13:45.670163 7f29149e4700  1 mon.magna102@0(leader).log v22026 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22026)
>2018-07-11 22:13:49.518125 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:13:49.670593 7f29149e4700  1 mon.magna102@0(leader).log v22027 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22027)
>2018-07-11 22:13:53.654585 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11059. Immutable memtables: 0.
>
>2018-07-11 22:13:53.654630 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1288] Syncing log #11056
>2018-07-11 22:13:53.671114 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.654621) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:13:53.671125 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1288] Flushing memtable with next log file: 11059
>
>2018-07-11 22:13:53.671152 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347233671131, "job": 1288, "event": "flush_started", "num_memtables": 1, "num_entries": 1040, "num_deletes": 251, "memory_usage": 1742128}
>2018-07-11 22:13:53.671155 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1288] Level-0 flush table #11060: started
>2018-07-11 22:13:53.721611 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347233721596, "cf_name": "default", "job": 1288, "event": "table_file_creation", "file_number": 11060, "file_size": 1651804, "table_properties": {"data_size": 1637667, "index_size": 7354, "filter_size": 5863, "raw_key_size": 10042, "raw_average_key_size": 20, "raw_value_size": 1625612, "raw_average_value_size": 3324, "num_data_blocks": 224, "num_entries": 489, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:13:53.721622 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1288] Level-0 flush table #11060: 1651804 bytes OK
>2018-07-11 22:13:53.780027 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.738256) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11060 started
>2018-07-11 22:13:53.780035 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.779990) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11060: memtable #1 done
>2018-07-11 22:13:53.780038 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.780005) EVENT_LOG_v1 {"time_micros": 1531347233779998, "job": 1288, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:13:53.780040 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.780016) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:13:53.780058 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1288] Try to delete WAL files size 1720501, prev total WAL file size 1720501, number of live WAL files 2.
>
>2018-07-11 22:13:53.780604 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:13:53.780665 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:13:53.780658) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:13:53.780672 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1289] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:13:53.780678 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1288 Base level 0, inputs: [11060(1613KB)], [11058(14MB)]
>
>2018-07-11 22:13:53.780691 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347233780682, "job": 1289, "event": "compaction_started", "files_L0": [11060], "files_L1": [11058], "score": -1, "input_data_size": 16754674}
>2018-07-11 22:13:53.955731 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1289] Generated table #11061: 2984 keys, 14838748 bytes
>2018-07-11 22:13:53.955780 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347233955753, "cf_name": "default", "job": 1289, "event": "table_file_creation", "file_number": 11061, "file_size": 14838748, "table_properties": {"data_size": 14723413, "index_size": 66459, "filter_size": 47949, "raw_key_size": 61442, "raw_average_key_size": 20, "raw_value_size": 14634000, "raw_average_value_size": 4904, "num_data_blocks": 1931, "num_entries": 2984, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:13:53.988887 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1289] Compacted 1@0 + 1@1 files to L1 => 14838748 bytes
>2018-07-11 22:13:54.022423 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:13:54.022357) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.5 rd, 71.3 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.4) out(14.2), read-write-amplify(19.1) write-amplify(9.0) OK, records in: 3494, records dropped: 510
>
>2018-07-11 22:13:54.022432 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:13:54.022378) EVENT_LOG_v1 {"time_micros": 1531347234022368, "job": 1289, "event": "compaction_finished", "compaction_time_micros": 208177, "output_level": 1, "num_output_files": 1, "total_output_size": 14838748, "num_input_records": 3494, "num_output_records": 2984, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:13:54.022851 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347234022846, "job": 1289, "event": "table_file_deletion", "file_number": 11060}
>2018-07-11 22:13:54.025620 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347234025617, "job": 1289, "event": "table_file_deletion", "file_number": 11058}
>2018-07-11 22:14:05.525423 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:14:05.673858 7f29149e4700  1 mon.magna102@0(leader).log v22028 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22028)
>2018-07-11 22:14:09.557303 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:15:09.595980 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:15:29.563201 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:15:29.714263 7f29149e4700  1 mon.magna102@0(leader).log v22029 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22029)
>2018-07-11 22:15:31.564139 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 6)
>2018-07-11 22:15:31.714461 7f29149e4700  1 mon.magna102@0(leader).log v22030 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22030)
>2018-07-11 22:15:36.665176 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11062. Immutable memtables: 0.
>
>2018-07-11 22:15:36.665264 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1290] Syncing log #11059
>2018-07-11 22:15:36.681708 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.665255) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:15:36.681720 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1290] Flushing memtable with next log file: 11062
>
>2018-07-11 22:15:36.681746 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347336681738, "job": 1290, "event": "flush_started", "num_memtables": 1, "num_entries": 636, "num_deletes": 250, "memory_usage": 859328}
>2018-07-11 22:15:36.681749 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1290] Level-0 flush table #11063: started
>2018-07-11 22:15:36.725155 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347336725142, "cf_name": "default", "job": 1290, "event": "table_file_creation", "file_number": 11063, "file_size": 504275, "table_properties": {"data_size": 499086, "index_size": 2074, "filter_size": 2196, "raw_key_size": 7064, "raw_average_key_size": 21, "raw_value_size": 493159, "raw_average_value_size": 1517, "num_data_blocks": 65, "num_entries": 325, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:15:36.725165 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1290] Level-0 flush table #11063: 504275 bytes OK
>2018-07-11 22:15:36.783521 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.741681) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11063 started
>2018-07-11 22:15:36.783529 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.783486) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11063: memtable #1 done
>2018-07-11 22:15:36.783532 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.783500) EVENT_LOG_v1 {"time_micros": 1531347336783494, "job": 1290, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:15:36.783534 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.783511) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:15:36.783553 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1290] Try to delete WAL files size 845768, prev total WAL file size 845768, number of live WAL files 2.
>
>2018-07-11 22:15:36.783844 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:15:36.783899 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:15:36.783892) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:15:36.783908 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1291] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:15:36.783914 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1290 Base level 0, inputs: [11063(492KB)], [11061(14MB)]
>
>2018-07-11 22:15:36.783927 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347336783918, "job": 1291, "event": "compaction_started", "files_L0": [11063], "files_L1": [11061], "score": -1, "input_data_size": 15343023}
>2018-07-11 22:15:36.950885 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1291] Generated table #11064: 2846 keys, 13936452 bytes
>2018-07-11 22:15:36.950934 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347336950907, "cf_name": "default", "job": 1291, "event": "table_file_creation", "file_number": 11064, "file_size": 13936452, "table_properties": {"data_size": 13829153, "index_size": 61433, "filter_size": 44939, "raw_key_size": 58287, "raw_average_key_size": 20, "raw_value_size": 13745306, "raw_average_value_size": 4829, "num_data_blocks": 1790, "num_entries": 2846, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:15:36.984064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1291] Compacted 1@0 + 1@1 files to L1 => 13936452 bytes
>2018-07-11 22:15:37.017645 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:15:37.017569) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.7 rd, 69.6 wr, level 1, files in(1, 1) out(1) MB in(0.5, 14.2) out(13.3), read-write-amplify(58.1) write-amplify(27.6) OK, records in: 3309, records dropped: 463
>
>2018-07-11 22:15:37.017655 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:15:37.017594) EVENT_LOG_v1 {"time_micros": 1531347337017583, "job": 1291, "event": "compaction_finished", "compaction_time_micros": 200120, "output_level": 1, "num_output_files": 1, "total_output_size": 13936452, "num_input_records": 3309, "num_output_records": 2846, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:15:37.017827 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347337017823, "job": 1291, "event": "table_file_deletion", "file_number": 11063}
>2018-07-11 22:15:37.020774 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347337020771, "job": 1291, "event": "table_file_deletion", "file_number": 11061}
>2018-07-11 22:15:37.020890 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:15:45.570561 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:15:45.718990 7f29149e4700  1 mon.magna102@0(leader).log v22031 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22031)
>2018-07-11 22:16:09.620465 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:17:05.606498 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:17:05.759108 7f29149e4700  1 mon.magna102@0(leader).log v22032 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22032)
>2018-07-11 22:17:09.642548 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:17:15.610917 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:17:15.759960 7f29149e4700  1 mon.magna102@0(leader).log v22033 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22033)
>2018-07-11 22:17:25.615344 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:17:25.769124 7f29149e4700  1 mon.magna102@0(leader).log v22034 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22034)
>2018-07-11 22:17:31.681026 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:17:31.803061 7f29149e4700  1 mon.magna102@0(leader).log v22035 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22035)
>2018-07-11 22:18:09.673262 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:18:51.654006 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:18:51.801488 7f29149e4700  1 mon.magna102@0(leader).log v22036 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22036)
>2018-07-11 22:18:55.655734 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:18:55.802037 7f29149e4700  1 mon.magna102@0(leader).log v22037 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22037)
>2018-07-11 22:19:09.661909 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:19:09.695699 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:19:09.812745 7f29149e4700  1 mon.magna102@0(leader).log v22038 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22038)
>2018-07-11 22:19:16.696272 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:19:16.796625 7f29149e4700  1 mon.magna102@0(leader).log v22039 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22039)
>2018-07-11 22:20:09.726669 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:20:31.698931 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:20:31.853660 7f29149e4700  1 mon.magna102@0(leader).log v22040 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22040)
>2018-07-11 22:20:35.700663 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:20:35.854225 7f29149e4700  1 mon.magna102@0(leader).log v22041 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22041)
>2018-07-11 22:20:45.705017 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:20:45.855127 7f29149e4700  1 mon.magna102@0(leader).log v22042 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22042)
>2018-07-11 22:20:51.770132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:20:51.889073 7f29149e4700  1 mon.magna102@0(leader).log v22043 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22043)
>2018-07-11 22:21:09.748420 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8795 MB, avail 861 GB
>2018-07-11 22:21:37.868666 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11065. Immutable memtables: 0.
>
>2018-07-11 22:21:37.868712 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1292] Syncing log #11062
>2018-07-11 22:21:37.885230 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.868702) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:21:37.885241 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1292] Flushing memtable with next log file: 11065
>
>2018-07-11 22:21:37.885267 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347697885259, "job": 1292, "event": "flush_started", "num_memtables": 1, "num_entries": 1634, "num_deletes": 251, "memory_usage": 3057128}
>2018-07-11 22:21:37.885271 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1292] Level-0 flush table #11066: started
>2018-07-11 22:21:37.944119 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347697944105, "cf_name": "default", "job": 1292, "event": "table_file_creation", "file_number": 11066, "file_size": 2903149, "table_properties": {"data_size": 2879474, "index_size": 12946, "filter_size": 9805, "raw_key_size": 13732, "raw_average_key_size": 20, "raw_value_size": 2860923, "raw_average_value_size": 4302, "num_data_blocks": 391, "num_entries": 665, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:21:37.944130 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1292] Level-0 flush table #11066: 2903149 bytes OK
>2018-07-11 22:21:37.994112 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.960628) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11066 started
>2018-07-11 22:21:37.994121 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.994074) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11066: memtable #1 done
>2018-07-11 22:21:37.994123 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.994089) EVENT_LOG_v1 {"time_micros": 1531347697994083, "job": 1292, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:21:37.994125 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.994102) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:21:37.994144 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1292] Try to delete WAL files size 3025286, prev total WAL file size 3025286, number of live WAL files 2.
>
>2018-07-11 22:21:37.994985 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:21:37.995210 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 22:21:37.995219 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 109724.9 total, 928.4 interval
>Cumulative writes: 136K writes, 637K keys, 136K commit groups, 1.0 writes per commit group, ingest: 1.04 GB, 0.01 MB/s
>Cumulative WAL: 136K writes, 136K syncs, 1.00 writes per sync, written: 1.04 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 5038 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.35 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.77 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        50       647    0.078       0      0
>  L1      1/1   13.29 MB   0.0     10.2     0.9      9.3       9.3     -0.0       0.0  10.8     72.5     66.4       144       644    0.224   2317K   281K
> Sum      2/2   16.06 MB   0.0     10.2     0.9      9.3      10.2      0.9       0.0  11.8     53.7     53.8       194      1291    0.151   2317K   281K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  12.1     58.3     56.9         1        10    0.140     18K   2972
>Uptime(secs): 109724.9 total, 109724.9 interval
>Flush(GB): cumulative 0.864, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.20 GB write, 0.10 MB/s write, 10.20 GB read, 0.10 MB/s read, 194.3 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.4 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.77 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        50       647    0.078       0      0
>  L1      1/1   13.29 MB   0.0     10.2     0.9      9.3       9.3     -0.0       0.0  10.8     72.5     66.4       144       644    0.224   2317K   281K
> Sum      2/2   16.06 MB   0.0     10.2     0.9      9.3      10.2      0.9       0.0  11.8     53.7     53.8       194      1291    0.151   2317K   281K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 109724.9 total, 109724.9 interval
>Flush(GB): cumulative 0.864, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.20 GB write, 0.10 MB/s write, 10.20 GB read, 0.10 MB/s read, 194.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 22:21:37.995242 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:21:37.995234) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:21:37.995249 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1293] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:21:37.995254 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1292 Base level 0, inputs: [11066(2835KB)], [11064(13MB)]
>
>2018-07-11 22:21:37.995268 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347697995258, "job": 1293, "event": "compaction_started", "files_L0": [11066], "files_L1": [11064], "score": -1, "input_data_size": 16839601}
>2018-07-11 22:21:38.178149 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1293] Generated table #11067: 2999 keys, 14905414 bytes
>2018-07-11 22:21:38.178213 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347698178171, "cf_name": "default", "job": 1293, "event": "table_file_creation", "file_number": 11067, "file_size": 14905414, "table_properties": {"data_size": 14789992, "index_size": 66401, "filter_size": 48094, "raw_key_size": 61718, "raw_average_key_size": 20, "raw_value_size": 14700404, "raw_average_value_size": 4901, "num_data_blocks": 1929, "num_entries": 2999, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:21:38.211335 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1293] Compacted 1@0 + 1@1 files to L1 => 14905414 bytes
>2018-07-11 22:21:38.244845 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:21:38.244778) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.9 rd, 69.0 wr, level 1, files in(1, 1) out(1) MB in(2.8, 13.3) out(14.2), read-write-amplify(10.9) write-amplify(5.1) OK, records in: 3511, records dropped: 512
>
>2018-07-11 22:21:38.244853 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:21:38.244799) EVENT_LOG_v1 {"time_micros": 1531347698244789, "job": 1293, "event": "compaction_finished", "compaction_time_micros": 216051, "output_level": 1, "num_output_files": 1, "total_output_size": 14905414, "num_input_records": 3511, "num_output_records": 2999, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:21:38.245560 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347698245556, "job": 1293, "event": "table_file_deletion", "file_number": 11066}
>2018-07-11 22:21:38.248146 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347698248143, "job": 1293, "event": "table_file_deletion", "file_number": 11064}
>2018-07-11 22:22:09.780527 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:22:11.743903 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:22:11.915599 7f29149e4700  1 mon.magna102@0(leader).log v22044 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22044)
>2018-07-11 22:22:15.745630 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:22:15.899475 7f29149e4700  1 mon.magna102@0(leader).log v22045 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22045)
>2018-07-11 22:22:29.751878 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:22:29.900644 7f29149e4700  1 mon.magna102@0(leader).log v22046 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22046)
>2018-07-11 22:22:36.784154 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:22:36.884532 7f29149e4700  1 mon.magna102@0(leader).log v22047 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22047)
>2018-07-11 22:23:09.803288 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:23:51.787898 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:23:51.940932 7f29149e4700  1 mon.magna102@0(leader).log v22048 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22048)
>2018-07-11 22:23:55.789541 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:23:55.949689 7f29149e4700  1 mon.magna102@0(leader).log v22049 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22049)
>2018-07-11 22:23:56.908470 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11068. Immutable memtables: 0.
>
>2018-07-11 22:23:56.908514 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1294] Syncing log #11065
>2018-07-11 22:23:56.924958 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:23:56.908504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:23:56.924969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1294] Flushing memtable with next log file: 11068
>
>2018-07-11 22:23:56.924996 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347836924988, "job": 1294, "event": "flush_started", "num_memtables": 1, "num_entries": 786, "num_deletes": 250, "memory_usage": 1199000}
>2018-07-11 22:23:56.924998 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1294] Level-0 flush table #11069: started
>2018-07-11 22:23:56.967154 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347836967140, "cf_name": "default", "job": 1294, "event": "table_file_creation", "file_number": 11069, "file_size": 704813, "table_properties": {"data_size": 698237, "index_size": 2850, "filter_size": 2807, "raw_key_size": 7607, "raw_average_key_size": 21, "raw_value_size": 691377, "raw_average_value_size": 1964, "num_data_blocks": 89, "num_entries": 352, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:23:56.967164 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1294] Level-0 flush table #11069: 704813 bytes OK
>2018-07-11 22:23:57.017161 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:23:56.983693) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11069 started
>2018-07-11 22:23:57.017169 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.017125) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11069: memtable #1 done
>2018-07-11 22:23:57.017171 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.017139) EVENT_LOG_v1 {"time_micros": 1531347837017133, "job": 1294, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:23:57.017195 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.017151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:23:57.017202 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1294] Try to delete WAL files size 1182532, prev total WAL file size 1182532, number of live WAL files 2.
>
>2018-07-11 22:23:57.017616 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:23:57.017677 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.017670) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:23:57.017683 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1295] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:23:57.017691 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1294 Base level 0, inputs: [11069(688KB)], [11067(14MB)]
>
>2018-07-11 22:23:57.017704 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347837017694, "job": 1295, "event": "compaction_started", "files_L0": [11069], "files_L1": [11067], "score": -1, "input_data_size": 15610227}
>2018-07-11 22:23:57.192890 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1295] Generated table #11070: 2906 keys, 14322806 bytes
>2018-07-11 22:23:57.192938 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347837192911, "cf_name": "default", "job": 1295, "event": "table_file_creation", "file_number": 11070, "file_size": 14322806, "table_properties": {"data_size": 14212986, "index_size": 62804, "filter_size": 46089, "raw_key_size": 59502, "raw_average_key_size": 20, "raw_value_size": 14127328, "raw_average_value_size": 4861, "num_data_blocks": 1830, "num_entries": 2906, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:23:57.226086 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1295] Compacted 1@0 + 1@1 files to L1 => 14322806 bytes
>2018-07-11 22:23:57.259615 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.259544) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.9 rd, 68.7 wr, level 1, files in(1, 1) out(1) MB in(0.7, 14.2) out(13.7), read-write-amplify(42.5) write-amplify(20.3) OK, records in: 3351, records dropped: 445
>
>2018-07-11 22:23:57.259624 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:23:57.259566) EVENT_LOG_v1 {"time_micros": 1531347837259556, "job": 1295, "event": "compaction_finished", "compaction_time_micros": 208365, "output_level": 1, "num_output_files": 1, "total_output_size": 14322806, "num_input_records": 3351, "num_output_records": 2906, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:23:57.259848 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347837259844, "job": 1295, "event": "table_file_deletion", "file_number": 11069}
>2018-07-11 22:23:57.262735 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531347837262732, "job": 1295, "event": "table_file_deletion", "file_number": 11067}
>2018-07-11 22:23:57.262856 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:24:09.827611 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:24:11.796396 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:24:11.954279 7f29149e4700  1 mon.magna102@0(leader).log v22050 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22050)
>2018-07-11 22:24:16.867274 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:24:16.946438 7f29149e4700  1 mon.magna102@0(leader).log v22051 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22051)
>2018-07-11 22:25:09.849903 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:25:31.830916 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:25:31.986246 7f29149e4700  1 mon.magna102@0(leader).log v22052 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22052)
>2018-07-11 22:25:35.832567 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:25:35.986669 7f29149e4700  1 mon.magna102@0(leader).log v22053 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22053)
>2018-07-11 22:25:55.841303 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:25:55.996495 7f29149e4700  1 mon.magna102@0(leader).log v22054 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22054)
>2018-07-11 22:26:09.880548 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:27:09.927450 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:27:11.874901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:27:12.019623 7f29149e4700  1 mon.magna102@0(leader).log v22055 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22055)
>2018-07-11 22:27:15.876546 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:27:16.028336 7f29149e4700  1 mon.magna102@0(leader).log v22056 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22056)
>2018-07-11 22:27:31.883505 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:27:32.037919 7f29149e4700  1 mon.magna102@0(leader).log v22057 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22057)
>2018-07-11 22:27:36.921122 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:27:37.021700 7f29149e4700  1 mon.magna102@0(leader).log v22058 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22058)
>2018-07-11 22:28:09.933440 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:28:55.920616 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 5,6)
>2018-07-11 22:28:56.086862 7f29149e4700  1 mon.magna102@0(leader).log v22059 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22059)
>2018-07-11 22:29:09.963914 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:29:11.927514 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:29:12.081086 7f29149e4700  1 mon.magna102@0(leader).log v22060 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22060)
>2018-07-11 22:29:16.964315 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:29:17.064850 7f29149e4700  1 mon.magna102@0(leader).log v22061 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22061)
>2018-07-11 22:29:20.065541 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11071. Immutable memtables: 0.
>
>2018-07-11 22:29:20.065585 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1296] Syncing log #11068
>2018-07-11 22:29:20.082080 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.065575) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:29:20.082091 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1296] Flushing memtable with next log file: 11071
>
>2018-07-11 22:29:20.082118 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160082110, "job": 1296, "event": "flush_started", "num_memtables": 1, "num_entries": 1486, "num_deletes": 251, "memory_usage": 2739560}
>2018-07-11 22:29:20.082122 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1296] Level-0 flush table #11072: started
>2018-07-11 22:29:20.149299 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160149285, "cf_name": "default", "job": 1296, "event": "table_file_creation", "file_number": 11072, "file_size": 2603759, "table_properties": {"data_size": 2582417, "index_size": 11565, "filter_size": 8853, "raw_key_size": 12783, "raw_average_key_size": 20, "raw_value_size": 2565501, "raw_average_value_size": 4137, "num_data_blocks": 350, "num_entries": 620, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:29:20.149310 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1296] Level-0 flush table #11072: 2603759 bytes OK
>2018-07-11 22:29:20.199243 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.165802) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11072 started
>2018-07-11 22:29:20.199251 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.199206) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11072: memtable #1 done
>2018-07-11 22:29:20.199254 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.199220) EVENT_LOG_v1 {"time_micros": 1531348160199214, "job": 1296, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:29:20.199268 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.199233) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:29:20.199275 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1296] Try to delete WAL files size 2710631, prev total WAL file size 2710631, number of live WAL files 2.
>
>2018-07-11 22:29:20.199973 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:29:20.200041 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.200031) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:29:20.200048 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1297] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:29:20.200054 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1296 Base level 0, inputs: [11072(2542KB)], [11070(13MB)]
>
>2018-07-11 22:29:20.200067 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160200058, "job": 1297, "event": "compaction_started", "files_L0": [11072], "files_L1": [11070], "score": -1, "input_data_size": 16926565}
>2018-07-11 22:29:20.374986 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1297] Generated table #11073: 3016 keys, 14976361 bytes
>2018-07-11 22:29:20.375033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160375007, "cf_name": "default", "job": 1297, "event": "table_file_creation", "file_number": 11073, "file_size": 14976361, "table_properties": {"data_size": 14860750, "index_size": 66413, "filter_size": 48271, "raw_key_size": 62038, "raw_average_key_size": 20, "raw_value_size": 14770915, "raw_average_value_size": 4897, "num_data_blocks": 1929, "num_entries": 3016, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:29:20.408036 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1297] Compacted 1@0 + 1@1 files to L1 => 14976361 bytes
>2018-07-11 22:29:20.441708 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.441640) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.4 rd, 72.0 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.7) out(14.3), read-write-amplify(12.3) write-amplify(5.8) OK, records in: 3526, records dropped: 510
>
>2018-07-11 22:29:20.441716 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:29:20.441661) EVENT_LOG_v1 {"time_micros": 1531348160441652, "job": 1297, "event": "compaction_finished", "compaction_time_micros": 207949, "output_level": 1, "num_output_files": 1, "total_output_size": 14976361, "num_input_records": 3526, "num_output_records": 3016, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:29:20.442340 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160442335, "job": 1297, "event": "table_file_deletion", "file_number": 11072}
>2018-07-11 22:29:20.445019 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348160445016, "job": 1297, "event": "table_file_deletion", "file_number": 11070}
>2018-07-11 22:30:09.988493 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:30:31.962705 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:30:32.132113 7f29149e4700  1 mon.magna102@0(leader).log v22062 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22062)
>2018-07-11 22:30:39.966049 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:30:40.117674 7f29149e4700  1 mon.magna102@0(leader).log v22063 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22063)
>2018-07-11 22:30:51.971028 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:30:52.118747 7f29149e4700  1 mon.magna102@0(leader).log v22064 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22064)
>2018-07-11 22:30:57.001788 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:30:57.102530 7f29149e4700  1 mon.magna102@0(leader).log v22065 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22065)
>2018-07-11 22:31:10.011700 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:32:10.033641 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:32:16.007284 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:32:16.167757 7f29149e4700  1 mon.magna102@0(leader).log v22066 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22066)
>2018-07-11 22:32:17.126514 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11074. Immutable memtables: 0.
>
>2018-07-11 22:32:17.126557 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1298] Syncing log #11071
>2018-07-11 22:32:17.143035 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.126547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:32:17.143046 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1298] Flushing memtable with next log file: 11074
>
>2018-07-11 22:32:17.143072 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337143064, "job": 1298, "event": "flush_started", "num_memtables": 1, "num_entries": 911, "num_deletes": 250, "memory_usage": 1471872}
>2018-07-11 22:32:17.143075 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1298] Level-0 flush table #11075: started
>2018-07-11 22:32:17.193550 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337193536, "cf_name": "default", "job": 1298, "event": "table_file_creation", "file_number": 11075, "file_size": 832116, "table_properties": {"data_size": 824603, "index_size": 3388, "filter_size": 3206, "raw_key_size": 7926, "raw_average_key_size": 21, "raw_value_size": 817127, "raw_average_value_size": 2220, "num_data_blocks": 106, "num_entries": 368, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:32:17.193561 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1298] Level-0 flush table #11075: 832116 bytes OK
>2018-07-11 22:32:17.243599 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.210105) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11075 started
>2018-07-11 22:32:17.243607 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.243563) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11075: memtable #1 done
>2018-07-11 22:32:17.243609 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.243577) EVENT_LOG_v1 {"time_micros": 1531348337243571, "job": 1298, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:32:17.243624 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.243589) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:32:17.243630 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1298] Try to delete WAL files size 1453641, prev total WAL file size 1453641, number of live WAL files 2.
>
>2018-07-11 22:32:17.244139 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:32:17.244333 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 22:32:17.244340 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 110364.1 total, 639.2 interval
>Cumulative writes: 137K writes, 640K keys, 137K commit groups, 1.0 writes per commit group, ingest: 1.04 GB, 0.01 MB/s
>Cumulative WAL: 137K writes, 137K syncs, 1.00 writes per sync, written: 1.04 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 690 writes, 3183 keys, 690 commit groups, 1.0 writes per commit group, ingest: 5.10 MB, 0.01 MB/s
>Interval WAL: 690 writes, 690 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   812.61 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       650    0.078       0      0
>  L1      1/1   14.28 MB   0.0     10.2     0.9      9.4       9.4     -0.0       0.0  10.8     72.5     66.4       145       647    0.224   2327K   282K
> Sum      2/2   15.08 MB   0.0     10.2     0.9      9.4      10.2      0.9       0.0  11.8     53.8     53.8       195      1297    0.150   2327K   282K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.7     55.9     54.8         1         6    0.140     10K   1467
>Uptime(secs): 110364.1 total, 110364.1 interval
>Flush(GB): cumulative 0.868, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.25 GB write, 0.10 MB/s write, 10.25 GB read, 0.10 MB/s read, 195.2 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   812.61 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       650    0.078       0      0
>  L1      1/1   14.28 MB   0.0     10.2     0.9      9.4       9.4     -0.0       0.0  10.8     72.5     66.4       145       647    0.224   2327K   282K
> Sum      2/2   15.08 MB   0.0     10.2     0.9      9.4      10.2      0.9       0.0  11.8     53.8     53.8       195      1297    0.150   2327K   282K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 110364.1 total, 110364.1 interval
>Flush(GB): cumulative 0.868, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.25 GB write, 0.10 MB/s write, 10.25 GB read, 0.10 MB/s read, 195.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 22:32:17.244363 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.244357) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:32:17.244368 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1299] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:32:17.244373 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1298 Base level 0, inputs: [11075(812KB)], [11073(14MB)]
>
>2018-07-11 22:32:17.244386 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337244378, "job": 1299, "event": "compaction_started", "files_L0": [11075], "files_L1": [11073], "score": -1, "input_data_size": 15808477}
>2018-07-11 22:32:17.410977 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1299] Generated table #11076: 2958 keys, 14651507 bytes
>2018-07-11 22:32:17.411025 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337410997, "cf_name": "default", "job": 1299, "event": "table_file_creation", "file_number": 11076, "file_size": 14651507, "table_properties": {"data_size": 14539458, "index_size": 64034, "filter_size": 47088, "raw_key_size": 60559, "raw_average_key_size": 20, "raw_value_size": 14452198, "raw_average_value_size": 4885, "num_data_blocks": 1866, "num_entries": 2958, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:32:17.444170 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1299] Compacted 1@0 + 1@1 files to L1 => 14651507 bytes
>2018-07-11 22:32:17.477696 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.477626) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.1 rd, 73.3 wr, level 1, files in(1, 1) out(1) MB in(0.8, 14.3) out(14.0), read-write-amplify(36.6) write-amplify(17.6) OK, records in: 3384, records dropped: 426
>
>2018-07-11 22:32:17.477705 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:32:17.477647) EVENT_LOG_v1 {"time_micros": 1531348337477638, "job": 1299, "event": "compaction_finished", "compaction_time_micros": 199768, "output_level": 1, "num_output_files": 1, "total_output_size": 14651507, "num_input_records": 3384, "num_output_records": 2958, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:32:17.477951 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337477947, "job": 1299, "event": "table_file_deletion", "file_number": 11075}
>2018-07-11 22:32:17.480816 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348337480813, "job": 1299, "event": "table_file_deletion", "file_number": 11073}
>2018-07-11 22:32:17.480928 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:32:20.009172 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:32:20.161941 7f29149e4700  1 mon.magna102@0(leader).log v22067 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22067)
>2018-07-11 22:32:36.016146 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:32:36.172787 7f29149e4700  1 mon.magna102@0(leader).log v22068 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22068)
>2018-07-11 22:33:10.066770 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:33:56.051104 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:33:56.204793 7f29149e4700  1 mon.magna102@0(leader).log v22069 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22069)
>2018-07-11 22:34:00.052748 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:34:00.205251 7f29149e4700  1 mon.magna102@0(leader).log v22070 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22070)
>2018-07-11 22:34:10.089085 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:34:16.059609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:34:16.214819 7f29149e4700  1 mon.magna102@0(leader).log v22071 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22071)
>2018-07-11 22:34:22.098500 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:34:22.215492 7f29149e4700  1 mon.magna102@0(leader).log v22072 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22072)
>2018-07-11 22:35:10.128237 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:35:36.094752 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:35:36.247017 7f29149e4700  1 mon.magna102@0(leader).log v22073 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22073)
>2018-07-11 22:35:40.096278 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:35:40.239131 7f29149e4700  1 mon.magna102@0(leader).log v22074 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22074)
>2018-07-11 22:35:56.103281 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:35:56.257058 7f29149e4700  1 mon.magna102@0(leader).log v22075 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22075)
>2018-07-11 22:36:02.141744 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:36:02.258925 7f29149e4700  1 mon.magna102@0(leader).log v22076 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22076)
>2018-07-11 22:36:10.150984 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:37:08.263958 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11077. Immutable memtables: 0.
>
>2018-07-11 22:37:08.264003 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1300] Syncing log #11074
>2018-07-11 22:37:08.281752 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.263993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:37:08.281763 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1300] Flushing memtable with next log file: 11077
>
>2018-07-11 22:37:08.281790 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628281781, "job": 1300, "event": "flush_started", "num_memtables": 1, "num_entries": 1358, "num_deletes": 251, "memory_usage": 2459960}
>2018-07-11 22:37:08.281793 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1300] Level-0 flush table #11078: started
>2018-07-11 22:37:08.340573 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628340558, "cf_name": "default", "job": 1300, "event": "table_file_creation", "file_number": 11078, "file_size": 2335965, "table_properties": {"data_size": 2316687, "index_size": 10346, "filter_size": 8008, "raw_key_size": 11989, "raw_average_key_size": 20, "raw_value_size": 2301170, "raw_average_value_size": 3953, "num_data_blocks": 314, "num_entries": 582, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:37:08.340596 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1300] Level-0 flush table #11078: 2335965 bytes OK
>2018-07-11 22:37:08.390573 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.357071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11078 started
>2018-07-11 22:37:08.390581 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.390534) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11078: memtable #1 done
>2018-07-11 22:37:08.390584 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.390549) EVENT_LOG_v1 {"time_micros": 1531348628390543, "job": 1300, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:37:08.390586 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.390562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:37:08.390592 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1300] Try to delete WAL files size 2432949, prev total WAL file size 2432949, number of live WAL files 2.
>
>2018-07-11 22:37:08.391213 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:37:08.391265 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.391258) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:37:08.391275 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1301] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:37:08.391281 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1300 Base level 0, inputs: [11078(2281KB)], [11076(13MB)]
>
>2018-07-11 22:37:08.391295 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628391285, "job": 1301, "event": "compaction_started", "files_L0": [11078], "files_L1": [11076], "score": -1, "input_data_size": 16987472}
>2018-07-11 22:37:08.582956 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1301] Generated table #11079: 3030 keys, 15023090 bytes
>2018-07-11 22:37:08.583007 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628582978, "cf_name": "default", "job": 1301, "event": "table_file_creation", "file_number": 11079, "file_size": 15023090, "table_properties": {"data_size": 14907336, "index_size": 66431, "filter_size": 48396, "raw_key_size": 62301, "raw_average_key_size": 20, "raw_value_size": 14817292, "raw_average_value_size": 4890, "num_data_blocks": 1929, "num_entries": 3030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:37:08.616140 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1301] Compacted 1@0 + 1@1 files to L1 => 15023090 bytes
>2018-07-11 22:37:08.649659 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.649592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.6 rd, 66.8 wr, level 1, files in(1, 1) out(1) MB in(2.2, 14.0) out(14.3), read-write-amplify(13.7) write-amplify(6.4) OK, records in: 3540, records dropped: 510
>
>2018-07-11 22:37:08.649669 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:37:08.649613) EVENT_LOG_v1 {"time_micros": 1531348628649604, "job": 1301, "event": "compaction_finished", "compaction_time_micros": 224828, "output_level": 1, "num_output_files": 1, "total_output_size": 15023090, "num_input_records": 3540, "num_output_records": 3030, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:37:08.650211 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628650207, "job": 1301, "event": "table_file_deletion", "file_number": 11078}
>2018-07-11 22:37:08.653062 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348628653059, "job": 1301, "event": "table_file_deletion", "file_number": 11076}
>2018-07-11 22:37:10.175135 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:37:16.138511 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:37:16.292647 7f29149e4700  1 mon.magna102@0(leader).log v22077 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22077)
>2018-07-11 22:37:20.140128 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:37:20.293109 7f29149e4700  1 mon.magna102@0(leader).log v22078 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22078)
>2018-07-11 22:37:36.147053 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:37:36.319379 7f29149e4700  1 mon.magna102@0(leader).log v22079 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22079)
>2018-07-11 22:38:10.197831 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:38:56.182086 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:38:56.334791 7f29149e4700  1 mon.magna102@0(leader).log v22080 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22080)
>2018-07-11 22:39:00.183516 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:39:00.335250 7f29149e4700  1 mon.magna102@0(leader).log v22081 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22081)
>2018-07-11 22:39:10.219079 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:39:16.190604 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:39:16.404435 7f29149e4700  1 mon.magna102@0(leader).log v22082 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22082)
>2018-07-11 22:40:10.250108 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:40:36.225723 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:40:36.378607 7f29149e4700  1 mon.magna102@0(leader).log v22083 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22083)
>2018-07-11 22:40:37.328968 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11080. Immutable memtables: 0.
>
>2018-07-11 22:40:37.329012 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1302] Syncing log #11077
>2018-07-11 22:40:37.345537 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.329002) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:40:37.345548 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1302] Flushing memtable with next log file: 11080
>
>2018-07-11 22:40:37.345574 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837345566, "job": 1302, "event": "flush_started", "num_memtables": 1, "num_entries": 1039, "num_deletes": 250, "memory_usage": 1756920}
>2018-07-11 22:40:37.345578 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1302] Level-0 flush table #11081: started
>2018-07-11 22:40:37.387730 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837387716, "cf_name": "default", "job": 1302, "event": "table_file_creation", "file_number": 11081, "file_size": 995140, "table_properties": {"data_size": 986483, "index_size": 4031, "filter_size": 3707, "raw_key_size": 8368, "raw_average_key_size": 21, "raw_value_size": 978237, "raw_average_value_size": 2508, "num_data_blocks": 126, "num_entries": 390, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:40:37.387740 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1302] Level-0 flush table #11081: 995140 bytes OK
>2018-07-11 22:40:37.437769 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.404262) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11081 started
>2018-07-11 22:40:37.437777 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.437733) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11081: memtable #1 done
>2018-07-11 22:40:37.437779 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.437747) EVENT_LOG_v1 {"time_micros": 1531348837437741, "job": 1302, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:40:37.437794 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.437759) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:40:37.437800 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1302] Try to delete WAL files size 1735929, prev total WAL file size 1735929, number of live WAL files 2.
>
>2018-07-11 22:40:37.438296 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:40:37.438347 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.438340) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:40:37.438357 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1303] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:40:37.438363 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1302 Base level 0, inputs: [11081(971KB)], [11079(14MB)]
>
>2018-07-11 22:40:37.438377 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837438367, "job": 1303, "event": "compaction_started", "files_L0": [11081], "files_L1": [11079], "score": -1, "input_data_size": 16018230}
>2018-07-11 22:40:37.621657 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1303] Generated table #11082: 3010 keys, 14970867 bytes
>2018-07-11 22:40:37.621697 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837621671, "cf_name": "default", "job": 1303, "event": "table_file_creation", "file_number": 11082, "file_size": 14970867, "table_properties": {"data_size": 14856563, "index_size": 65301, "filter_size": 48076, "raw_key_size": 61616, "raw_average_key_size": 20, "raw_value_size": 14767675, "raw_average_value_size": 4906, "num_data_blocks": 1903, "num_entries": 3010, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:40:37.655024 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1303] Compacted 1@0 + 1@1 files to L1 => 14970867 bytes
>2018-07-11 22:40:37.688507 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.688440) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.9 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.3) out(14.3), read-write-amplify(31.1) write-amplify(15.0) OK, records in: 3420, records dropped: 410
>
>2018-07-11 22:40:37.688516 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:40:37.688461) EVENT_LOG_v1 {"time_micros": 1531348837688452, "job": 1303, "event": "compaction_finished", "compaction_time_micros": 216630, "output_level": 1, "num_output_files": 1, "total_output_size": 14970867, "num_input_records": 3420, "num_output_records": 3010, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:40:37.688793 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837688789, "job": 1303, "event": "table_file_deletion", "file_number": 11081}
>2018-07-11 22:40:37.691681 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531348837691678, "job": 1303, "event": "table_file_deletion", "file_number": 11079}
>2018-07-11 22:40:37.691798 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:40:40.227558 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:40:40.397781 7f29149e4700  1 mon.magna102@0(leader).log v22084 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22084)
>2018-07-11 22:40:56.234546 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)
>2018-07-11 22:40:56.407582 7f29149e4700  1 mon.magna102@0(leader).log v22085 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22085)
>2018-07-11 22:41:02.296340 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 5,6 (REQUEST_SLOW)
>2018-07-11 22:41:02.424887 7f29149e4700  1 mon.magna102@0(leader).log v22086 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22086)
>2018-07-11 22:41:10.276599 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:42:10.305655 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:42:16.269722 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 5 (REQUEST_SLOW)
>2018-07-11 22:42:16.423103 7f29149e4700  1 mon.magna102@0(leader).log v22087 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22087)
>2018-07-11 22:42:20.271396 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 5)
>2018-07-11 22:42:20.433117 7f29149e4700  1 mon.magna102@0(leader).log v22088 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22088)
>2018-07-11 22:42:21.508256 7f29149e4700  1 mon.magna102@0(leader).log v22089 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22089)
>2018-07-11 22:42:23.675115 7f29149e4700  1 mon.magna102@0(leader).log v22090 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22090)
>2018-07-11 22:42:25.842248 7f29149e4700  1 mon.magna102@0(leader).log v22091 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22091)
>2018-07-11 22:42:27.009247 7f29149e4700  1 mon.magna102@0(leader).log v22092 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22092)
>2018-07-11 22:42:42.281032 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:42:42.452340 7f29149e4700  1 mon.magna102@0(leader).log v22093 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22093)
>2018-07-11 22:42:47.318849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:42:47.411111 7f29149e4700  1 mon.magna102@0(leader).log v22094 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22094)
>2018-07-11 22:43:10.329299 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:43:54.312528 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:43:54.475496 7f29149e4700  1 mon.magna102@0(leader).log v22095 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22095)
>2018-07-11 22:44:02.316261 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:44:02.468015 7f29149e4700  1 mon.magna102@0(leader).log v22096 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22096)
>2018-07-11 22:44:10.360101 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:44:20.324012 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 22:44:20.486038 7f29149e4700  1 mon.magna102@0(leader).log v22097 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22097)
>2018-07-11 22:44:27.367415 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:44:27.486690 7f29149e4700  1 mon.magna102@0(leader).log v22098 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22098)
>2018-07-11 22:44:42.464450 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11083. Immutable memtables: 0.
>
>2018-07-11 22:44:42.464495 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1304] Syncing log #11080
>2018-07-11 22:44:42.480985 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.464485) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:44:42.480996 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1304] Flushing memtable with next log file: 11083
>
>2018-07-11 22:44:42.481010 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082481003, "job": 1304, "event": "flush_started", "num_memtables": 1, "num_entries": 1237, "num_deletes": 251, "memory_usage": 2197344}
>2018-07-11 22:44:42.481013 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1304] Level-0 flush table #11084: started
>2018-07-11 22:44:42.539873 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082539859, "cf_name": "default", "job": 1304, "event": "table_file_creation", "file_number": 11084, "file_size": 2092370, "table_properties": {"data_size": 2075056, "index_size": 9163, "filter_size": 7231, "raw_key_size": 11328, "raw_average_key_size": 20, "raw_value_size": 2060820, "raw_average_value_size": 3740, "num_data_blocks": 279, "num_entries": 551, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:44:42.539884 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1304] Level-0 flush table #11084: 2092370 bytes OK
>2018-07-11 22:44:42.598252 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.556411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11084 started
>2018-07-11 22:44:42.598260 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.598215) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11084: memtable #1 done
>2018-07-11 22:44:42.598263 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.598230) EVENT_LOG_v1 {"time_micros": 1531349082598224, "job": 1304, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:44:42.598265 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.598243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:44:42.598271 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1304] Try to delete WAL files size 2172416, prev total WAL file size 2172416, number of live WAL files 2.
>
>2018-07-11 22:44:42.598831 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:44:42.599014 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 22:44:42.599020 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 111109.5 total, 745.4 interval
>Cumulative writes: 137K writes, 644K keys, 137K commit groups, 1.0 writes per commit group, ingest: 1.05 GB, 0.01 MB/s
>Cumulative WAL: 137K writes, 137K syncs, 1.00 writes per sync, written: 1.05 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 816 writes, 3634 keys, 816 commit groups, 1.0 writes per commit group, ingest: 6.05 MB, 0.01 MB/s
>Interval WAL: 816 writes, 816 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.00 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       653    0.078       0      0
>  L1      1/1   14.28 MB   0.0     10.3     0.9      9.4       9.4     -0.0       0.0  10.8     72.5     66.4       145       650    0.224   2337K   284K
> Sum      2/2   16.27 MB   0.0     10.3     0.9      9.4      10.3      0.9       0.0  11.8     53.8     53.8       196      1303    0.150   2337K   284K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.2     54.7     56.1         1         6    0.142     10K   1346
>Uptime(secs): 111109.5 total, 111109.5 interval
>Flush(GB): cumulative 0.873, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.29 GB write, 0.09 MB/s write, 10.29 GB read, 0.09 MB/s read, 196.0 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.00 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       653    0.078       0      0
>  L1      1/1   14.28 MB   0.0     10.3     0.9      9.4       9.4     -0.0       0.0  10.8     72.5     66.4       145       650    0.224   2337K   284K
> Sum      2/2   16.27 MB   0.0     10.3     0.9      9.4      10.3      0.9       0.0  11.8     53.8     53.8       196      1303    0.150   2337K   284K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 111109.5 total, 111109.5 interval
>Flush(GB): cumulative 0.873, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.29 GB write, 0.09 MB/s write, 10.29 GB read, 0.09 MB/s read, 196.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 22:44:42.599041 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.599034) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:44:42.599046 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1305] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:44:42.599051 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1304 Base level 0, inputs: [11084(2043KB)], [11082(14MB)]
>
>2018-07-11 22:44:42.599063 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082599054, "job": 1305, "event": "compaction_started", "files_L0": [11084], "files_L1": [11082], "score": -1, "input_data_size": 17063237}
>2018-07-11 22:44:42.782301 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1305] Generated table #11085: 3051 keys, 15087690 bytes
>2018-07-11 22:44:42.782330 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082782316, "cf_name": "default", "job": 1305, "event": "table_file_creation", "file_number": 11085, "file_size": 15087690, "table_properties": {"data_size": 14971713, "index_size": 66474, "filter_size": 48576, "raw_key_size": 62697, "raw_average_key_size": 20, "raw_value_size": 14881346, "raw_average_value_size": 4877, "num_data_blocks": 1930, "num_entries": 3051, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:44:42.823809 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1305] Compacted 1@0 + 1@1 files to L1 => 15087690 bytes
>2018-07-11 22:44:42.865686 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.865614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.9 rd, 67.1 wr, level 1, files in(1, 1) out(1) MB in(2.0, 14.3) out(14.4), read-write-amplify(15.4) write-amplify(7.2) OK, records in: 3561, records dropped: 510
>
>2018-07-11 22:44:42.865695 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:44:42.865635) EVENT_LOG_v1 {"time_micros": 1531349082865625, "job": 1305, "event": "compaction_finished", "compaction_time_micros": 224730, "output_level": 1, "num_output_files": 1, "total_output_size": 15087690, "num_input_records": 3561, "num_output_records": 3051, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:44:42.866162 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082866158, "job": 1305, "event": "table_file_deletion", "file_number": 11084}
>2018-07-11 22:44:42.869033 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349082869030, "job": 1305, "event": "table_file_deletion", "file_number": 11082}
>2018-07-11 22:45:10.376353 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:45:34.356773 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:45:34.504581 7f29149e4700  1 mon.magna102@0(leader).log v22099 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22099)
>2018-07-11 22:45:52.364768 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:45:52.514316 7f29149e4700  1 mon.magna102@0(leader).log v22100 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22100)
>2018-07-11 22:46:02.369154 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:46:02.523492 7f29149e4700  1 mon.magna102@0(leader).log v22101 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22101)
>2018-07-11 22:46:07.406681 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:46:07.507288 7f29149e4700  1 mon.magna102@0(leader).log v22102 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22102)
>2018-07-11 22:46:10.415761 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:47:10.437612 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:47:14.400595 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:47:14.546786 7f29149e4700  1 mon.magna102@0(leader).log v22103 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22103)
>2018-07-11 22:47:32.408538 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:47:32.556532 7f29149e4700  1 mon.magna102@0(leader).log v22104 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22104)
>2018-07-11 22:47:44.413605 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 22:47:44.567082 7f29149e4700  1 mon.magna102@0(leader).log v22105 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22105)
>2018-07-11 22:47:52.450802 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:47:52.567822 7f29149e4700  1 mon.magna102@0(leader).log v22106 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22106)
>2018-07-11 22:48:10.460519 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-11 22:48:57.548938 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11086. Immutable memtables: 0.
>
>2018-07-11 22:48:57.548982 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1306] Syncing log #11083
>2018-07-11 22:48:57.565467 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.548972) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:48:57.565479 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1306] Flushing memtable with next log file: 11086
>
>2018-07-11 22:48:57.565492 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337565485, "job": 1306, "event": "flush_started", "num_memtables": 1, "num_entries": 1208, "num_deletes": 250, "memory_usage": 2124536}
>2018-07-11 22:48:57.565495 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1306] Level-0 flush table #11087: started
>2018-07-11 22:48:57.616011 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337615997, "cf_name": "default", "job": 1306, "event": "table_file_creation", "file_number": 11087, "file_size": 1185316, "table_properties": {"data_size": 1175264, "index_size": 4830, "filter_size": 4302, "raw_key_size": 8889, "raw_average_key_size": 21, "raw_value_size": 1166074, "raw_average_value_size": 2803, "num_data_blocks": 151, "num_entries": 416, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:48:57.616022 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1306] Level-0 flush table #11087: 1185316 bytes OK
>2018-07-11 22:48:57.666062 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.632612) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11087 started
>2018-07-11 22:48:57.666070 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.666026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11087: memtable #1 done
>2018-07-11 22:48:57.666072 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.666041) EVENT_LOG_v1 {"time_micros": 1531349337666034, "job": 1306, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:48:57.666074 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.666052) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:48:57.666081 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1306] Try to delete WAL files size 2100910, prev total WAL file size 2100910, number of live WAL files 2.
>
>2018-07-11 22:48:57.666639 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:48:57.666692 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.666685) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:48:57.666701 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1307] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:48:57.666706 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1306 Base level 0, inputs: [11087(1157KB)], [11085(14MB)]
>
>2018-07-11 22:48:57.666719 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337666710, "job": 1307, "event": "compaction_started", "files_L0": [11087], "files_L1": [11085], "score": -1, "input_data_size": 16273006}
>2018-07-11 22:48:57.858478 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1307] Generated table #11088: 3080 keys, 15383596 bytes
>2018-07-11 22:48:57.858515 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337858499, "cf_name": "default", "job": 1307, "event": "table_file_creation", "file_number": 11088, "file_size": 15383596, "table_properties": {"data_size": 15266365, "index_size": 66946, "filter_size": 49358, "raw_key_size": 63039, "raw_average_key_size": 20, "raw_value_size": 15175334, "raw_average_value_size": 4927, "num_data_blocks": 1951, "num_entries": 3080, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:48:57.899953 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1307] Compacted 1@0 + 1@1 files to L1 => 15383596 bytes
>2018-07-11 22:48:57.933493 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.933425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.8 rd, 66.0 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.4) out(14.7), read-write-amplify(26.7) write-amplify(13.0) OK, records in: 3467, records dropped: 387
>
>2018-07-11 22:48:57.933502 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:48:57.933446) EVENT_LOG_v1 {"time_micros": 1531349337933437, "job": 1307, "event": "compaction_finished", "compaction_time_micros": 233218, "output_level": 1, "num_output_files": 1, "total_output_size": 15383596, "num_input_records": 3467, "num_output_records": 3080, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:48:57.933782 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337933778, "job": 1307, "event": "table_file_deletion", "file_number": 11087}
>2018-07-11 22:48:57.936668 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349337936665, "job": 1307, "event": "table_file_deletion", "file_number": 11085}
>2018-07-11 22:48:57.936722 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:48:58.548837 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:48:58.742641 7f29149e4700  1 mon.magna102@0(leader).log v22107 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22107)
>2018-07-11 22:49:10.484949 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:49:12.452541 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:49:12.602199 7f29149e4700  1 mon.magna102@0(leader).log v22108 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22108)
>2018-07-11 22:49:26.458709 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:49:26.611647 7f29149e4700  1 mon.magna102@0(leader).log v22109 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22109)
>2018-07-11 22:49:32.516613 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:49:32.637437 7f29149e4700  1 mon.magna102@0(leader).log v22110 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22110)
>2018-07-11 22:50:10.516393 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:50:38.490359 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:50:38.660566 7f29149e4700  1 mon.magna102@0(leader).log v22111 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22111)
>2018-07-11 22:50:53.629607 7f29149e4700  1 mon.magna102@0(leader).log v22112 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22112)
>2018-07-11 22:50:56.498453 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:50:56.655160 7f29149e4700  1 mon.magna102@0(leader).log v22113 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22113)
>2018-07-11 22:51:06.502638 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:51:06.656008 7f29149e4700  1 mon.magna102@0(leader).log v22114 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22114)
>2018-07-11 22:51:10.539320 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:51:12.539625 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:51:12.656661 7f29149e4700  1 mon.magna102@0(leader).log v22115 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22115)
>2018-07-11 22:52:10.570091 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:52:24.536278 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:52:24.713040 7f29149e4700  1 mon.magna102@0(leader).log v22116 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22116)
>2018-07-11 22:52:24.796785 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11089. Immutable memtables: 0.
>
>2018-07-11 22:52:24.796830 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1308] Syncing log #11086
>2018-07-11 22:52:24.813346 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.796821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:52:24.813357 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1308] Flushing memtable with next log file: 11089
>
>2018-07-11 22:52:24.813371 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349544813363, "job": 1308, "event": "flush_started", "num_memtables": 1, "num_entries": 1064, "num_deletes": 251, "memory_usage": 1806840}
>2018-07-11 22:52:24.813374 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1308] Level-0 flush table #11090: started
>2018-07-11 22:52:24.863903 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349544863888, "cf_name": "default", "job": 1308, "event": "table_file_creation", "file_number": 11090, "file_size": 1715586, "table_properties": {"data_size": 1701065, "index_size": 7553, "filter_size": 6048, "raw_key_size": 10225, "raw_average_key_size": 20, "raw_value_size": 1688741, "raw_average_value_size": 3391, "num_data_blocks": 230, "num_entries": 498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 22:52:24.863913 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1308] Level-0 flush table #11090: 1715586 bytes OK
>2018-07-11 22:52:24.913893 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.880455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11090 started
>2018-07-11 22:52:24.913901 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.913856) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11090: memtable #1 done
>2018-07-11 22:52:24.913903 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.913871) EVENT_LOG_v1 {"time_micros": 1531349544913864, "job": 1308, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:52:24.913907 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.913883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:52:24.913913 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1308] Try to delete WAL files size 1784905, prev total WAL file size 1784905, number of live WAL files 2.
>
>2018-07-11 22:52:24.914367 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:52:24.914419 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:52:24.914412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 22:52:24.914429 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1309] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:52:24.914435 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1308 Base level 0, inputs: [11090(1675KB)], [11088(14MB)]
>
>2018-07-11 22:52:24.914447 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349544914438, "job": 1309, "event": "compaction_started", "files_L0": [11090], "files_L1": [11088], "score": -1, "input_data_size": 17099182}
>2018-07-11 22:52:25.109794 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1309] Generated table #11091: 3068 keys, 15126966 bytes
>2018-07-11 22:52:25.109845 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349545109816, "cf_name": "default", "job": 1309, "event": "table_file_creation", "file_number": 11091, "file_size": 15126966, "table_properties": {"data_size": 15010857, "index_size": 66491, "filter_size": 48691, "raw_key_size": 63017, "raw_average_key_size": 20, "raw_value_size": 14920235, "raw_average_value_size": 4863, "num_data_blocks": 1930, "num_entries": 3068, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:52:25.139491 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1309] Compacted 1@0 + 1@1 files to L1 => 15126966 bytes
>2018-07-11 22:52:25.173024 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:52:25.172957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.0 rd, 67.2 wr, level 1, files in(1, 1) out(1) MB in(1.6, 14.7) out(14.4), read-write-amplify(18.8) write-amplify(8.8) OK, records in: 3578, records dropped: 510
>
>2018-07-11 22:52:25.173032 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:52:25.172979) EVENT_LOG_v1 {"time_micros": 1531349545172969, "job": 1309, "event": "compaction_finished", "compaction_time_micros": 225027, "output_level": 1, "num_output_files": 1, "total_output_size": 15126966, "num_input_records": 3578, "num_output_records": 3068, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:52:25.173439 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349545173435, "job": 1309, "event": "table_file_deletion", "file_number": 11090}
>2018-07-11 22:52:25.176335 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349545176332, "job": 1309, "event": "table_file_deletion", "file_number": 11088}
>2018-07-11 22:52:32.540119 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:52:32.715896 7f29149e4700  1 mon.magna102@0(leader).log v22117 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22117)
>2018-07-11 22:52:52.548650 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:52:52.702058 7f29149e4700  1 mon.magna102@0(leader).log v22118 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22118)
>2018-07-11 22:52:57.585293 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:52:57.685837 7f29149e4700  1 mon.magna102@0(leader).log v22119 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22119)
>2018-07-11 22:53:10.595144 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:53:58.577340 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:53:58.733276 7f29149e4700  1 mon.magna102@0(leader).log v22120 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22120)
>2018-07-11 22:54:10.617195 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:54:16.585373 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:54:16.759689 7f29149e4700  1 mon.magna102@0(leader).log v22121 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22121)
>2018-07-11 22:54:28.590322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 22:54:28.760688 7f29149e4700  1 mon.magna102@0(leader).log v22122 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22122)
>2018-07-11 22:54:37.627643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:54:37.728075 7f29149e4700  1 mon.magna102@0(leader).log v22123 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22123)
>2018-07-11 22:55:10.648078 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:55:44.623384 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:55:44.775895 7f29149e4700  1 mon.magna102@0(leader).log v22124 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22124)
>2018-07-11 22:55:56.628829 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:55:56.785214 7f29149e4700  1 mon.magna102@0(leader).log v22125 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22125)
>2018-07-11 22:56:08.633783 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 22:56:08.786174 7f29149e4700  1 mon.magna102@0(leader).log v22126 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22126)
>2018-07-11 22:56:10.677797 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:56:16.637516 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:56:16.813256 7f29149e4700  1 mon.magna102@0(leader).log v22127 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22127)
>2018-07-11 22:57:10.691883 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:57:17.784557 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11092. Immutable memtables: 0.
>
>2018-07-11 22:57:17.784600 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1310] Syncing log #11089
>2018-07-11 22:57:17.801060 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.784590) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 22:57:17.801071 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1310] Flushing memtable with next log file: 11092
>
>2018-07-11 22:57:17.801098 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349837801089, "job": 1310, "event": "flush_started", "num_memtables": 1, "num_entries": 1365, "num_deletes": 250, "memory_usage": 2466936}
>2018-07-11 22:57:17.801101 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1310] Level-0 flush table #11093: started
>2018-07-11 22:57:17.852827 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349837852813, "cf_name": "default", "job": 1310, "event": "table_file_creation", "file_number": 11093, "file_size": 1382982, "table_properties": {"data_size": 1371505, "index_size": 5638, "filter_size": 4919, "raw_key_size": 9452, "raw_average_key_size": 21, "raw_value_size": 1361345, "raw_average_value_size": 3066, "num_data_blocks": 176, "num_entries": 444, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 22:57:17.852837 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1310] Level-0 flush table #11093: 1382982 bytes OK
>2018-07-11 22:57:17.902851 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.869361) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11093 started
>2018-07-11 22:57:17.902859 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.902815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11093: memtable #1 done
>2018-07-11 22:57:17.902861 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.902829) EVENT_LOG_v1 {"time_micros": 1531349837902823, "job": 1310, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 22:57:17.902863 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.902841) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 22:57:17.902882 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1310] Try to delete WAL files size 2440235, prev total WAL file size 2440235, number of live WAL files 2.
>
>2018-07-11 22:57:17.903614 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:57:17.903809 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 22:57:17.903816 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 111864.8 total, 755.3 interval
>Cumulative writes: 138K writes, 648K keys, 138K commit groups, 1.0 writes per commit group, ingest: 1.06 GB, 0.01 MB/s
>Cumulative WAL: 138K writes, 138K syncs, 1.00 writes per sync, written: 1.06 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 818 writes, 3637 keys, 818 commit groups, 1.0 writes per commit group, ingest: 6.03 MB, 0.01 MB/s
>Interval WAL: 818 writes, 818 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.32 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       656    0.078       0      0
>  L1      1/1   14.43 MB   0.0     10.3     0.9      9.5       9.5     -0.0       0.0  10.8     72.5     66.4       146       653    0.224   2348K   285K
> Sum      2/2   15.75 MB   0.0     10.3     0.9      9.5      10.3      0.9       0.0  11.8     53.8     53.8       197      1309    0.150   2348K   285K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.6     54.3     53.7         1         6    0.148     10K   1407
>Uptime(secs): 111864.8 total, 111864.8 interval
>Flush(GB): cumulative 0.877, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.34 GB write, 0.09 MB/s write, 10.34 GB read, 0.09 MB/s read, 196.9 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.32 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       656    0.078       0      0
>  L1      1/1   14.43 MB   0.0     10.3     0.9      9.5       9.5     -0.0       0.0  10.8     72.5     66.4       146       653    0.224   2348K   285K
> Sum      2/2   15.75 MB   0.0     10.3     0.9      9.5      10.3      0.9       0.0  11.8     53.8     53.8       197      1309    0.150   2348K   285K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 111864.8 total, 111864.8 interval
>Flush(GB): cumulative 0.877, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.34 GB write, 0.09 MB/s write, 10.34 GB read, 0.09 MB/s read, 196.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 22:57:17.903837 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:57:17.903830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 22:57:17.903842 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1311] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 22:57:17.903847 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1310 Base level 0, inputs: [11093(1350KB)], [11091(14MB)]
>
>2018-07-11 22:57:17.903859 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349837903850, "job": 1311, "event": "compaction_started", "files_L0": [11093], "files_L1": [11091], "score": -1, "input_data_size": 16509948}
>2018-07-11 22:57:18.086891 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1311] Generated table #11094: 3144 keys, 15750294 bytes
>2018-07-11 22:57:18.086936 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349838086908, "cf_name": "default", "job": 1311, "event": "table_file_creation", "file_number": 11094, "file_size": 15750294, "table_properties": {"data_size": 15630407, "index_size": 68453, "filter_size": 50507, "raw_key_size": 64340, "raw_average_key_size": 20, "raw_value_size": 15537412, "raw_average_value_size": 4941, "num_data_blocks": 1995, "num_entries": 3144, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 22:57:18.120082 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1311] Compacted 1@0 + 1@1 files to L1 => 15750294 bytes
>2018-07-11 22:57:18.153620 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:57:18.153550) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.4 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.4) out(15.0), read-write-amplify(23.3) write-amplify(11.4) OK, records in: 3512, records dropped: 368
>
>2018-07-11 22:57:18.153629 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-22:57:18.153571) EVENT_LOG_v1 {"time_micros": 1531349838153562, "job": 1311, "event": "compaction_finished", "compaction_time_micros": 216205, "output_level": 1, "num_output_files": 1, "total_output_size": 15750294, "num_input_records": 3512, "num_output_records": 3144, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 22:57:18.153988 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349838153984, "job": 1311, "event": "table_file_deletion", "file_number": 11093}
>2018-07-11 22:57:18.156862 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531349838156859, "job": 1311, "event": "table_file_deletion", "file_number": 11091}
>2018-07-11 22:57:18.156975 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 22:57:24.667262 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:57:24.821499 7f29149e4700  1 mon.magna102@0(leader).log v22128 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22128)
>2018-07-11 22:57:36.672592 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:57:36.847461 7f29149e4700  1 mon.magna102@0(leader).log v22129 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22129)
>2018-07-11 22:57:52.679465 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:57:52.848859 7f29149e4700  1 mon.magna102@0(leader).log v22130 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22130)
>2018-07-11 22:57:57.752394 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:57:57.849317 7f29149e4700  1 mon.magna102@0(leader).log v22131 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22131)
>2018-07-11 22:58:10.716718 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:59:04.710732 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 22:59:04.863782 7f29149e4700  1 mon.magna102@0(leader).log v22132 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22132)
>2018-07-11 22:59:10.747348 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 22:59:16.715999 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 22:59:16.873246 7f29149e4700  1 mon.magna102@0(leader).log v22133 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22133)
>2018-07-11 22:59:34.723596 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 22:59:34.892516 7f29149e4700  1 mon.magna102@0(leader).log v22134 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22134)
>2018-07-11 22:59:42.768008 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 22:59:42.901701 7f29149e4700  1 mon.magna102@0(leader).log v22135 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22135)
>2018-07-11 23:00:04.870314 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11095. Immutable memtables: 0.
>
>2018-07-11 23:00:04.870359 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1312] Syncing log #11092
>2018-07-11 23:00:04.886921 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.870349) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:00:04.886932 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1312] Flushing memtable with next log file: 11095
>
>2018-07-11 23:00:04.886946 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350004886939, "job": 1312, "event": "flush_started", "num_memtables": 1, "num_entries": 908, "num_deletes": 251, "memory_usage": 1457176}
>2018-07-11 23:00:04.886949 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1312] Level-0 flush table #11096: started
>2018-07-11 23:00:04.945792 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350004945777, "cf_name": "default", "job": 1312, "event": "table_file_creation", "file_number": 11096, "file_size": 1383757, "table_properties": {"data_size": 1371729, "index_size": 6105, "filter_size": 5003, "raw_key_size": 9263, "raw_average_key_size": 20, "raw_value_size": 1361102, "raw_average_value_size": 3011, "num_data_blocks": 186, "num_entries": 452, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:00:04.945802 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1312] Level-0 flush table #11096: 1383757 bytes OK
>2018-07-11 23:00:04.995863 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.962340) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11096 started
>2018-07-11 23:00:04.995872 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.995827) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11096: memtable #1 done
>2018-07-11 23:00:04.995874 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.995842) EVENT_LOG_v1 {"time_micros": 1531350004995836, "job": 1312, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:00:04.995876 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.995854) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:00:04.995883 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1312] Try to delete WAL files size 1438127, prev total WAL file size 1438127, number of live WAL files 2.
>
>2018-07-11 23:00:04.996273 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:00:04.996325 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:00:04.996318) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:00:04.996334 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1313] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:00:04.996340 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1312 Base level 0, inputs: [11096(1351KB)], [11094(15MB)]
>
>2018-07-11 23:00:04.996352 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350004996343, "job": 1313, "event": "compaction_started", "files_L0": [11096], "files_L1": [11094], "score": -1, "input_data_size": 17134051}
>2018-07-11 23:00:05.179903 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1313] Generated table #11097: 3086 keys, 15150822 bytes
>2018-07-11 23:00:05.179953 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350005179925, "cf_name": "default", "job": 1313, "event": "table_file_creation", "file_number": 11097, "file_size": 15150822, "table_properties": {"data_size": 15034615, "index_size": 66505, "filter_size": 48775, "raw_key_size": 63356, "raw_average_key_size": 20, "raw_value_size": 14943729, "raw_average_value_size": 4842, "num_data_blocks": 1930, "num_entries": 3086, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:00:05.221423 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1313] Compacted 1@0 + 1@1 files to L1 => 15150822 bytes
>2018-07-11 23:00:05.254952 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:00:05.254882) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.1 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(1.3, 15.0) out(14.4), read-write-amplify(23.3) write-amplify(10.9) OK, records in: 3596, records dropped: 510
>
>2018-07-11 23:00:05.254962 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:00:05.254903) EVENT_LOG_v1 {"time_micros": 1531350005254893, "job": 1313, "event": "compaction_finished", "compaction_time_micros": 225053, "output_level": 1, "num_output_files": 1, "total_output_size": 15150822, "num_input_records": 3596, "num_output_records": 3086, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:00:05.255302 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350005255298, "job": 1313, "event": "table_file_deletion", "file_number": 11096}
>2018-07-11 23:00:05.258269 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350005258266, "job": 1313, "event": "table_file_deletion", "file_number": 11094}
>2018-07-11 23:00:10.780678 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:00:48.756277 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:00:48.910210 7f29149e4700  1 mon.magna102@0(leader).log v22136 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22136)
>2018-07-11 23:01:02.762508 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:01:02.919894 7f29149e4700  1 mon.magna102@0(leader).log v22137 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22137)
>2018-07-11 23:01:10.803635 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:01:14.767703 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:01:14.930509 7f29149e4700  1 mon.magna102@0(leader).log v22138 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22138)
>2018-07-11 23:01:22.814236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:01:22.947915 7f29149e4700  1 mon.magna102@0(leader).log v22139 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22139)
>2018-07-11 23:02:10.826225 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:02:28.799814 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:02:28.954041 7f29149e4700  1 mon.magna102@0(leader).log v22140 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22140)
>2018-07-11 23:02:40.805279 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:02:40.955030 7f29149e4700  1 mon.magna102@0(leader).log v22141 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22141)
>2018-07-11 23:02:58.812857 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:02:58.981442 7f29149e4700  1 mon.magna102@0(leader).log v22142 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22142)
>2018-07-11 23:03:07.873405 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:03:07.982152 7f29149e4700  1 mon.magna102@0(leader).log v22143 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22143)
>2018-07-11 23:03:10.857300 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:04:10.879443 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:04:14.846346 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:04:14.996633 7f29149e4700  1 mon.magna102@0(leader).log v22144 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22144)
>2018-07-11 23:04:26.851587 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:04:27.014407 7f29149e4700  1 mon.magna102@0(leader).log v22145 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22145)
>2018-07-11 23:04:40.857611 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:04:41.015548 7f29149e4700  1 mon.magna102@0(leader).log v22146 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22146)
>2018-07-11 23:04:47.891937 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:04:48.034038 7f29149e4700  1 mon.magna102@0(leader).log v22147 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22147)
>2018-07-11 23:05:10.902217 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:05:37.987869 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11098. Immutable memtables: 0.
>
>2018-07-11 23:05:37.987913 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1314] Syncing log #11095
>2018-07-11 23:05:38.004360 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:05:37.987904) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:05:38.004372 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1314] Flushing memtable with next log file: 11098
>
>2018-07-11 23:05:38.004386 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338004378, "job": 1314, "event": "flush_started", "num_memtables": 1, "num_entries": 1513, "num_deletes": 250, "memory_usage": 2802056}
>2018-07-11 23:05:38.004389 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1314] Level-0 flush table #11099: started
>2018-07-11 23:05:38.063242 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338063228, "cf_name": "default", "job": 1314, "event": "table_file_creation", "file_number": 11099, "file_size": 1564123, "table_properties": {"data_size": 1551390, "index_size": 6342, "filter_size": 5471, "raw_key_size": 9913, "raw_average_key_size": 21, "raw_value_size": 1540397, "raw_average_value_size": 3298, "num_data_blocks": 198, "num_entries": 467, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:05:38.063253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1314] Level-0 flush table #11099: 1564123 bytes OK
>2018-07-11 23:05:38.113304 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.079800) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11099 started
>2018-07-11 23:05:38.113312 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.113266) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11099: memtable #1 done
>2018-07-11 23:05:38.113315 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.113281) EVENT_LOG_v1 {"time_micros": 1531350338113274, "job": 1314, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:05:38.113317 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.113293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:05:38.113323 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1314] Try to delete WAL files size 2772615, prev total WAL file size 2772615, number of live WAL files 2.
>
>2018-07-11 23:05:38.114042 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:05:38.114094 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.114087) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:05:38.114103 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1315] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:05:38.114109 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1314 Base level 0, inputs: [11099(1527KB)], [11097(14MB)]
>
>2018-07-11 23:05:38.114122 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338114113, "job": 1315, "event": "compaction_started", "files_L0": [11099], "files_L1": [11097], "score": -1, "input_data_size": 16714945}
>2018-07-11 23:05:38.305684 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1315] Generated table #11100: 3205 keys, 16092564 bytes
>2018-07-11 23:05:38.305733 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338305705, "cf_name": "default", "job": 1315, "event": "table_file_creation", "file_number": 11100, "file_size": 16092564, "table_properties": {"data_size": 15970155, "index_size": 69891, "filter_size": 51591, "raw_key_size": 65580, "raw_average_key_size": 20, "raw_value_size": 15875286, "raw_average_value_size": 4953, "num_data_blocks": 2037, "num_entries": 3205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:05:38.347186 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1315] Compacted 1@0 + 1@1 files to L1 => 16092564 bytes
>2018-07-11 23:05:38.380722 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.380656) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.7 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.4) out(15.3), read-write-amplify(21.0) write-amplify(10.3) OK, records in: 3553, records dropped: 348
>
>2018-07-11 23:05:38.380731 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:05:38.380677) EVENT_LOG_v1 {"time_micros": 1531350338380668, "job": 1315, "event": "compaction_finished", "compaction_time_micros": 233041, "output_level": 1, "num_output_files": 1, "total_output_size": 16092564, "num_input_records": 3553, "num_output_records": 3205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:05:38.381092 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338381088, "job": 1315, "event": "table_file_deletion", "file_number": 11099}
>2018-07-11 23:05:38.383978 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350338383975, "job": 1315, "event": "table_file_deletion", "file_number": 11097}
>2018-07-11 23:05:38.384011 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:05:54.890143 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:05:55.042283 7f29149e4700  1 mon.magna102@0(leader).log v22148 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22148)
>2018-07-11 23:06:06.895420 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:06:07.051606 7f29149e4700  1 mon.magna102@0(leader).log v22149 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22149)
>2018-07-11 23:06:10.935036 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:06:20.901436 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:06:21.052855 7f29149e4700  1 mon.magna102@0(leader).log v22150 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22150)
>2018-07-11 23:06:27.956085 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:06:28.045098 7f29149e4700  1 mon.magna102@0(leader).log v22151 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22151)
>2018-07-11 23:07:10.957514 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:07:34.934060 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:07:35.084471 7f29149e4700  1 mon.magna102@0(leader).log v22152 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22152)
>2018-07-11 23:07:46.939302 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:07:47.093831 7f29149e4700  1 mon.magna102@0(leader).log v22153 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22153)
>2018-07-11 23:07:47.160907 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11101. Immutable memtables: 0.
>
>2018-07-11 23:07:47.160952 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1316] Syncing log #11098
>2018-07-11 23:07:47.177473 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.160942) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:07:47.177485 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1316] Flushing memtable with next log file: 11101
>
>2018-07-11 23:07:47.177511 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467177503, "job": 1316, "event": "flush_started", "num_memtables": 1, "num_entries": 759, "num_deletes": 251, "memory_usage": 1136336}
>2018-07-11 23:07:47.177514 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1316] Level-0 flush table #11102: started
>2018-07-11 23:07:47.229256 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467229242, "cf_name": "default", "job": 1316, "event": "table_file_creation", "file_number": 11102, "file_size": 1077381, "table_properties": {"data_size": 1067711, "index_size": 4722, "filter_size": 4028, "raw_key_size": 8343, "raw_average_key_size": 20, "raw_value_size": 1058711, "raw_average_value_size": 2594, "num_data_blocks": 144, "num_entries": 408, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:07:47.229266 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1316] Level-0 flush table #11102: 1077381 bytes OK
>2018-07-11 23:07:47.287590 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.245775) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11102 started
>2018-07-11 23:07:47.287598 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.287554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11102: memtable #1 done
>2018-07-11 23:07:47.287600 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.287569) EVENT_LOG_v1 {"time_micros": 1531350467287562, "job": 1316, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:07:47.287602 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.287580) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:07:47.287622 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1316] Try to delete WAL files size 1119970, prev total WAL file size 1119970, number of live WAL files 2.
>
>2018-07-11 23:07:47.288033 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:07:47.288241 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 23:07:47.288247 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 112494.1 total, 629.4 interval
>Cumulative writes: 139K writes, 651K keys, 139K commit groups, 1.0 writes per commit group, ingest: 1.06 GB, 0.01 MB/s
>Cumulative WAL: 139K writes, 139K syncs, 1.00 writes per sync, written: 1.06 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 688 writes, 3180 keys, 688 commit groups, 1.0 writes per commit group, ingest: 5.08 MB, 0.01 MB/s
>Interval WAL: 688 writes, 688 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.03 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       659    0.078       0      0
>  L1      1/1   15.35 MB   0.0     10.4     0.9      9.5       9.5     -0.0       0.0  10.8     72.5     66.4       147       656    0.224   2359K   286K
> Sum      2/2   16.37 MB   0.0     10.4     0.9      9.5      10.4      0.9       0.0  11.8     53.8     53.8       198      1315    0.150   2359K   286K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.7     53.8     54.5         1         6    0.149     10K   1226
>Uptime(secs): 112494.1 total, 112494.1 interval
>Flush(GB): cumulative 0.881, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.39 GB write, 0.09 MB/s write, 10.39 GB read, 0.09 MB/s read, 197.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.03 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       659    0.078       0      0
>  L1      1/1   15.35 MB   0.0     10.4     0.9      9.5       9.5     -0.0       0.0  10.8     72.5     66.4       147       656    0.224   2359K   286K
> Sum      2/2   16.37 MB   0.0     10.4     0.9      9.5      10.4      0.9       0.0  11.8     53.8     53.8       198      1315    0.150   2359K   286K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 112494.1 total, 112494.1 interval
>Flush(GB): cumulative 0.881, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.39 GB write, 0.09 MB/s write, 10.39 GB read, 0.09 MB/s read, 197.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 23:07:47.288268 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.288262) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:07:47.288273 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1317] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:07:47.288279 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1316 Base level 0, inputs: [11102(1052KB)], [11100(15MB)]
>
>2018-07-11 23:07:47.288291 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467288282, "job": 1317, "event": "compaction_started", "files_L0": [11102], "files_L1": [11100], "score": -1, "input_data_size": 17169945}
>2018-07-11 23:07:47.471642 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1317] Generated table #11103: 3103 keys, 15197944 bytes
>2018-07-11 23:07:47.471689 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467471662, "cf_name": "default", "job": 1317, "event": "table_file_creation", "file_number": 11103, "file_size": 15197944, "table_properties": {"data_size": 15081629, "index_size": 66490, "filter_size": 48898, "raw_key_size": 63676, "raw_average_key_size": 20, "raw_value_size": 14990516, "raw_average_value_size": 4830, "num_data_blocks": 1929, "num_entries": 3103, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:07:47.504844 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1317] Compacted 1@0 + 1@1 files to L1 => 15197944 bytes
>2018-07-11 23:07:47.546692 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.546624) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.3 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.3) out(14.5), read-write-amplify(30.0) write-amplify(14.1) OK, records in: 3613, records dropped: 510
>
>2018-07-11 23:07:47.546701 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:07:47.546646) EVENT_LOG_v1 {"time_micros": 1531350467546636, "job": 1317, "event": "compaction_finished", "compaction_time_micros": 216537, "output_level": 1, "num_output_files": 1, "total_output_size": 15197944, "num_input_records": 3613, "num_output_records": 3103, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:07:47.547000 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467546996, "job": 1317, "event": "table_file_deletion", "file_number": 11102}
>2018-07-11 23:07:47.550030 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350467550028, "job": 1317, "event": "table_file_deletion", "file_number": 11100}
>2018-07-11 23:08:04.947218 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:08:05.098596 7f29149e4700  1 mon.magna102@0(leader).log v22154 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22154)
>2018-07-11 23:08:10.982192 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:08:12.982357 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:08:13.099400 7f29149e4700  1 mon.magna102@0(leader).log v22155 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22155)
>2018-07-11 23:09:11.004886 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:09:24.982447 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:09:25.131173 7f29149e4700  1 mon.magna102@0(leader).log v22156 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22156)
>2018-07-11 23:09:30.985159 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:09:31.131755 7f29149e4700  1 mon.magna102@0(leader).log v22157 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22157)
>2018-07-11 23:09:46.992138 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:09:47.149791 7f29149e4700  1 mon.magna102@0(leader).log v22158 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22158)
>2018-07-11 23:09:53.034643 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:09:53.151628 7f29149e4700  1 mon.magna102@0(leader).log v22159 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22159)
>2018-07-11 23:10:11.036308 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:10:59.023782 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:10:59.174722 7f29149e4700  1 mon.magna102@0(leader).log v22160 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22160)
>2018-07-11 23:11:11.029278 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:11:11.058722 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:11:11.175764 7f29149e4700  1 mon.magna102@0(leader).log v22161 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22161)
>2018-07-11 23:11:29.036872 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:11:29.185488 7f29149e4700  1 mon.magna102@0(leader).log v22162 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22162)
>2018-07-11 23:11:38.077589 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:11:38.177961 7f29149e4700  1 mon.magna102@0(leader).log v22163 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22163)
>2018-07-11 23:12:11.098128 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:12:39.067751 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:12:39.242024 7f29149e4700  1 mon.magna102@0(leader).log v22164 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22164)
>2018-07-11 23:12:57.075843 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:12:57.226780 7f29149e4700  1 mon.magna102@0(leader).log v22165 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22165)
>2018-07-11 23:13:11.110886 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:13:15.083609 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:13:15.237780 7f29149e4700  1 mon.magna102@0(leader).log v22166 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22166)
>2018-07-11 23:13:23.121496 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:13:23.238563 7f29149e4700  1 mon.magna102@0(leader).log v22167 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22167)
>2018-07-11 23:13:58.216509 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11104. Immutable memtables: 0.
>
>2018-07-11 23:13:58.216586 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1318] Syncing log #11101
>2018-07-11 23:13:58.232878 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.216576) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:13:58.232890 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1318] Flushing memtable with next log file: 11104
>
>2018-07-11 23:13:58.232903 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838232896, "job": 1318, "event": "flush_started", "num_memtables": 1, "num_entries": 1662, "num_deletes": 250, "memory_usage": 3135088}
>2018-07-11 23:13:58.232906 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1318] Level-0 flush table #11105: started
>2018-07-11 23:13:58.283415 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838283401, "cf_name": "default", "job": 1318, "event": "table_file_creation", "file_number": 11105, "file_size": 1753096, "table_properties": {"data_size": 1739042, "index_size": 7083, "filter_size": 6051, "raw_key_size": 10415, "raw_average_key_size": 21, "raw_value_size": 1727169, "raw_average_value_size": 3510, "num_data_blocks": 221, "num_entries": 492, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:13:58.283426 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1318] Level-0 flush table #11105: 1753096 bytes OK
>2018-07-11 23:13:58.333473 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.299962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11105 started
>2018-07-11 23:13:58.333481 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.333436) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11105: memtable #1 done
>2018-07-11 23:13:58.333483 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.333450) EVENT_LOG_v1 {"time_micros": 1531350838333444, "job": 1318, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:13:58.333485 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.333463) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:13:58.333491 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1318] Try to delete WAL files size 3103150, prev total WAL file size 3103150, number of live WAL files 2.
>
>2018-07-11 23:13:58.334232 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:13:58.334285 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.334278) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:13:58.334294 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1319] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:13:58.334300 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1318 Base level 0, inputs: [11105(1712KB)], [11103(14MB)]
>
>2018-07-11 23:13:58.334312 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838334303, "job": 1319, "event": "compaction_started", "files_L0": [11105], "files_L1": [11103], "score": -1, "input_data_size": 16951040}
>2018-07-11 23:13:58.525754 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1319] Generated table #11106: 3266 keys, 16459002 bytes
>2018-07-11 23:13:58.525800 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838525772, "cf_name": "default", "job": 1319, "event": "table_file_creation", "file_number": 11106, "file_size": 16459002, "table_properties": {"data_size": 16334019, "index_size": 71332, "filter_size": 52724, "raw_key_size": 66820, "raw_average_key_size": 20, "raw_value_size": 16237276, "raw_average_value_size": 4971, "num_data_blocks": 2079, "num_entries": 3266, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:13:58.558933 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1319] Compacted 1@0 + 1@1 files to L1 => 16459002 bytes
>2018-07-11 23:13:58.592462 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.592394) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.5 rd, 73.3 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.5) out(15.7), read-write-amplify(19.1) write-amplify(9.4) OK, records in: 3595, records dropped: 329
>
>2018-07-11 23:13:58.592471 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:13:58.592416) EVENT_LOG_v1 {"time_micros": 1531350838592406, "job": 1319, "event": "compaction_finished", "compaction_time_micros": 224603, "output_level": 1, "num_output_files": 1, "total_output_size": 16459002, "num_input_records": 3595, "num_output_records": 3266, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:13:58.592878 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838592874, "job": 1319, "event": "table_file_deletion", "file_number": 11105}
>2018-07-11 23:13:58.595857 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350838595842, "job": 1319, "event": "table_file_deletion", "file_number": 11103}
>2018-07-11 23:13:58.595951 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:14:11.145086 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:14:25.114382 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:14:25.288065 7f29149e4700  1 mon.magna102@0(leader).log v22168 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22168)
>2018-07-11 23:14:41.121409 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:14:41.297742 7f29149e4700  1 mon.magna102@0(leader).log v22169 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22169)
>2018-07-11 23:14:51.125594 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:14:51.298617 7f29149e4700  1 mon.magna102@0(leader).log v22170 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22170)
>2018-07-11 23:14:58.184896 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:14:58.290835 7f29149e4700  1 mon.magna102@0(leader).log v22171 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22171)
>2018-07-11 23:15:11.166732 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:15:29.278032 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11107. Immutable memtables: 0.
>
>2018-07-11 23:15:29.278076 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1320] Syncing log #11104
>2018-07-11 23:15:29.294612 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.278067) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:15:29.294623 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1320] Flushing memtable with next log file: 11107
>
>2018-07-11 23:15:29.294637 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929294629, "job": 1320, "event": "flush_started", "num_memtables": 1, "num_entries": 610, "num_deletes": 251, "memory_usage": 806752}
>2018-07-11 23:15:29.294640 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1320] Level-0 flush table #11108: started
>2018-07-11 23:15:29.336701 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929336686, "cf_name": "default", "job": 1320, "event": "table_file_creation", "file_number": 11108, "file_size": 763438, "table_properties": {"data_size": 756144, "index_size": 3338, "filter_size": 3037, "raw_key_size": 7423, "raw_average_key_size": 20, "raw_value_size": 748766, "raw_average_value_size": 2057, "num_data_blocks": 102, "num_entries": 364, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:15:29.336712 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1320] Level-0 flush table #11108: 763438 bytes OK
>2018-07-11 23:15:29.386811 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.353400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11108 started
>2018-07-11 23:15:29.386819 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.386775) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11108: memtable #1 done
>2018-07-11 23:15:29.386821 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.386789) EVENT_LOG_v1 {"time_micros": 1531350929386783, "job": 1320, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:15:29.386823 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.386801) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:15:29.386829 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1320] Try to delete WAL files size 792897, prev total WAL file size 792897, number of live WAL files 2.
>
>2018-07-11 23:15:29.387063 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:15:29.387114 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.387107) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:15:29.387123 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1321] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:15:29.387129 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1320 Base level 0, inputs: [11108(745KB)], [11106(15MB)]
>
>2018-07-11 23:15:29.387141 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929387132, "job": 1321, "event": "compaction_started", "files_L0": [11108], "files_L1": [11106], "score": -1, "input_data_size": 17222440}
>2018-07-11 23:15:29.570892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1321] Generated table #11109: 3120 keys, 15254115 bytes
>2018-07-11 23:15:29.570936 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929570909, "cf_name": "default", "job": 1321, "event": "table_file_creation", "file_number": 11109, "file_size": 15254115, "table_properties": {"data_size": 15137639, "index_size": 66501, "filter_size": 49048, "raw_key_size": 63996, "raw_average_key_size": 20, "raw_value_size": 15046280, "raw_average_value_size": 4822, "num_data_blocks": 1929, "num_entries": 3120, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:15:29.604046 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1321] Compacted 1@0 + 1@1 files to L1 => 15254115 bytes
>2018-07-11 23:15:29.637600 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.637530) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.4 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.7) out(14.5), read-write-amplify(42.5) write-amplify(20.0) OK, records in: 3630, records dropped: 510
>
>2018-07-11 23:15:29.637608 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:15:29.637552) EVENT_LOG_v1 {"time_micros": 1531350929637542, "job": 1321, "event": "compaction_finished", "compaction_time_micros": 216889, "output_level": 1, "num_output_files": 1, "total_output_size": 15254115, "num_input_records": 3630, "num_output_records": 3120, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:15:29.637817 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929637813, "job": 1321, "event": "table_file_deletion", "file_number": 11108}
>2018-07-11 23:15:29.640879 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531350929640876, "job": 1321, "event": "table_file_deletion", "file_number": 11106}
>2018-07-11 23:16:05.157525 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:16:05.307431 7f29149e4700  1 mon.magna102@0(leader).log v22172 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22172)
>2018-07-11 23:16:11.200671 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:16:17.162766 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:16:17.334763 7f29149e4700  1 mon.magna102@0(leader).log v22173 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22173)
>2018-07-11 23:16:35.170408 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:16:35.344502 7f29149e4700  1 mon.magna102@0(leader).log v22174 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22174)
>2018-07-11 23:16:43.219343 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:16:43.336917 7f29149e4700  1 mon.magna102@0(leader).log v22175 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22175)
>2018-07-11 23:17:11.222035 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:17:49.202811 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:17:49.359477 7f29149e4700  1 mon.magna102@0(leader).log v22176 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22176)
>2018-07-11 23:18:01.208484 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:18:01.360567 7f29149e4700  1 mon.magna102@0(leader).log v22177 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22177)
>2018-07-11 23:18:11.252765 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:18:15.214366 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:18:15.386736 7f29149e4700  1 mon.magna102@0(leader).log v22178 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22178)
>2018-07-11 23:18:23.263630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:18:23.397373 7f29149e4700  1 mon.magna102@0(leader).log v22179 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22179)
>2018-07-11 23:19:11.275326 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:19:29.246846 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:19:29.402028 7f29149e4700  1 mon.magna102@0(leader).log v22180 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22180)
>2018-07-11 23:19:41.252491 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:19:41.404240 7f29149e4700  1 mon.magna102@0(leader).log v22181 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22181)
>2018-07-11 23:19:59.259843 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 9 (REQUEST_SLOW)
>2018-07-11 23:19:59.430659 7f29149e4700  1 mon.magna102@0(leader).log v22182 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22182)
>2018-07-11 23:20:08.306241 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 4,9 (REQUEST_SLOW)
>2018-07-11 23:20:08.439956 7f29149e4700  1 mon.magna102@0(leader).log v22183 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22183)
>2018-07-11 23:20:11.306829 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:20:26.825013 7f29149e4700  1 mon.magna102@0(leader).log v22184 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22184)
>2018-07-11 23:20:29.273506 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:20:29.425580 7f29149e4700  1 mon.magna102@0(leader).log v22185 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22185)
>2018-07-11 23:20:31.274621 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:20:31.425907 7f29149e4700  1 mon.magna102@0(leader).log v22186 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22186)
>2018-07-11 23:20:36.977745 7f29149e4700  1 mon.magna102@0(leader).log v22187 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22187)
>2018-07-11 23:21:11.330103 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:22:11.352161 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:22:18.419735 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11110. Immutable memtables: 0.
>
>2018-07-11 23:22:18.419778 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1322] Syncing log #11107
>2018-07-11 23:22:18.436331 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.419769) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:22:18.436342 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1322] Flushing memtable with next log file: 11110
>
>2018-07-11 23:22:18.436369 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338436361, "job": 1322, "event": "flush_started", "num_memtables": 1, "num_entries": 1820, "num_deletes": 250, "memory_usage": 3457728}
>2018-07-11 23:22:18.436372 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1322] Level-0 flush table #11111: started
>2018-07-11 23:22:18.486778 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338486764, "cf_name": "default", "job": 1322, "event": "table_file_creation", "file_number": 11111, "file_size": 1940170, "table_properties": {"data_size": 1924687, "index_size": 7910, "filter_size": 6653, "raw_key_size": 11052, "raw_average_key_size": 21, "raw_value_size": 1911733, "raw_average_value_size": 3655, "num_data_blocks": 247, "num_entries": 523, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:22:18.486789 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1322] Level-0 flush table #11111: 1940170 bytes OK
>2018-07-11 23:22:18.536902 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.503407) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11111 started
>2018-07-11 23:22:18.536910 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.536865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11111: memtable #1 done
>2018-07-11 23:22:18.536912 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.536880) EVENT_LOG_v1 {"time_micros": 1531351338536874, "job": 1322, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:22:18.536914 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.536892) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:22:18.536933 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1322] Try to delete WAL files size 3422521, prev total WAL file size 3422521, number of live WAL files 2.
>
>2018-07-11 23:22:18.537822 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:22:18.538017 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 23:22:18.538024 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 113365.4 total, 871.2 interval
>Cumulative writes: 140K writes, 655K keys, 140K commit groups, 1.0 writes per commit group, ingest: 1.07 GB, 0.01 MB/s
>Cumulative WAL: 140K writes, 140K syncs, 1.00 writes per sync, written: 1.07 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 946 writes, 4092 keys, 946 commit groups, 1.0 writes per commit group, ingest: 6.98 MB, 0.01 MB/s
>Interval WAL: 946 writes, 946 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.85 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       662    0.078       0      0
>  L1      1/1   14.55 MB   0.0     10.4     0.9      9.6       9.5     -0.0       0.0  10.8     72.5     66.4       147       659    0.224   2370K   288K
> Sum      2/2   16.40 MB   0.0     10.4     0.9      9.6      10.4      0.9       0.0  11.8     53.8     53.8       199      1321    0.150   2370K   288K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.5     57.5     57.6         1         6    0.142     10K   1349
>Uptime(secs): 113365.4 total, 113365.4 interval
>Flush(GB): cumulative 0.885, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.43 GB write, 0.09 MB/s write, 10.43 GB read, 0.09 MB/s read, 198.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.85 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.6        51       662    0.078       0      0
>  L1      1/1   14.55 MB   0.0     10.4     0.9      9.6       9.5     -0.0       0.0  10.8     72.5     66.4       147       659    0.224   2370K   288K
> Sum      2/2   16.40 MB   0.0     10.4     0.9      9.6      10.4      0.9       0.0  11.8     53.8     53.8       199      1321    0.150   2370K   288K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 113365.4 total, 113365.4 interval
>Flush(GB): cumulative 0.885, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.43 GB write, 0.09 MB/s write, 10.43 GB read, 0.09 MB/s read, 198.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 23:22:18.538049 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.538043) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:22:18.538054 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1323] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:22:18.538060 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1322 Base level 0, inputs: [11111(1894KB)], [11109(14MB)]
>
>2018-07-11 23:22:18.538074 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338538064, "job": 1323, "event": "compaction_started", "files_L0": [11111], "files_L1": [11109], "score": -1, "input_data_size": 17194285}
>2018-07-11 23:22:18.729309 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1323] Generated table #11112: 3330 keys, 16832177 bytes
>2018-07-11 23:22:18.729347 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338729330, "cf_name": "default", "job": 1323, "event": "table_file_creation", "file_number": 11112, "file_size": 16832177, "table_properties": {"data_size": 16704525, "index_size": 72836, "filter_size": 53889, "raw_key_size": 68117, "raw_average_key_size": 20, "raw_value_size": 16605821, "raw_average_value_size": 4986, "num_data_blocks": 2123, "num_entries": 3330, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:22:18.762424 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1323] Compacted 1@0 + 1@1 files to L1 => 16832177 bytes
>2018-07-11 23:22:18.795956 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.795890) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.6 rd, 75.0 wr, level 1, files in(1, 1) out(1) MB in(1.9, 14.5) out(16.1), read-write-amplify(17.5) write-amplify(8.7) OK, records in: 3643, records dropped: 313
>
>2018-07-11 23:22:18.795965 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:22:18.795911) EVENT_LOG_v1 {"time_micros": 1531351338795902, "job": 1323, "event": "compaction_finished", "compaction_time_micros": 224334, "output_level": 1, "num_output_files": 1, "total_output_size": 16832177, "num_input_records": 3643, "num_output_records": 3330, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:22:18.796449 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338796444, "job": 1323, "event": "table_file_deletion", "file_number": 11111}
>2018-07-11 23:22:18.799353 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351338799350, "job": 1323, "event": "table_file_deletion", "file_number": 11109}
>2018-07-11 23:22:18.799419 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:23:11.384555 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:23:13.485174 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11113. Immutable memtables: 0.
>
>2018-07-11 23:23:13.485250 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1324] Syncing log #11110
>2018-07-11 23:23:13.501824 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.485241) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:23:13.501836 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1324] Flushing memtable with next log file: 11113
>
>2018-07-11 23:23:13.501849 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393501842, "job": 1324, "event": "flush_started", "num_memtables": 1, "num_entries": 452, "num_deletes": 251, "memory_usage": 436840}
>2018-07-11 23:23:13.501852 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1324] Level-0 flush table #11114: started
>2018-07-11 23:23:13.544015 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393544002, "cf_name": "default", "job": 1324, "event": "table_file_creation", "file_number": 11114, "file_size": 412835, "table_properties": {"data_size": 408073, "index_size": 1889, "filter_size": 1954, "raw_key_size": 6372, "raw_average_key_size": 20, "raw_value_size": 402482, "raw_average_value_size": 1281, "num_data_blocks": 58, "num_entries": 314, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:23:13.544025 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1324] Level-0 flush table #11114: 412835 bytes OK
>2018-07-11 23:23:13.594088 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.560588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11114 started
>2018-07-11 23:23:13.594096 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.594054) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11114: memtable #1 done
>2018-07-11 23:23:13.594099 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.594068) EVENT_LOG_v1 {"time_micros": 1531351393594062, "job": 1324, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:23:13.594101 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.594080) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:23:13.594107 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1324] Try to delete WAL files size 425853, prev total WAL file size 425853, number of live WAL files 2.
>
>2018-07-11 23:23:13.594278 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:23:13.594328 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.594321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:23:13.594337 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1325] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:23:13.594343 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1324 Base level 0, inputs: [11114(403KB)], [11112(16MB)]
>
>2018-07-11 23:23:13.594356 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393594346, "job": 1325, "event": "compaction_started", "files_L0": [11114], "files_L1": [11112], "score": -1, "input_data_size": 17245012}
>2018-07-11 23:23:13.769791 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1325] Generated table #11115: 3136 keys, 15269756 bytes
>2018-07-11 23:23:13.769839 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393769812, "cf_name": "default", "job": 1325, "event": "table_file_creation", "file_number": 11115, "file_size": 15269756, "table_properties": {"data_size": 15153206, "index_size": 66512, "filter_size": 49111, "raw_key_size": 64293, "raw_average_key_size": 20, "raw_value_size": 15061624, "raw_average_value_size": 4802, "num_data_blocks": 1929, "num_entries": 3136, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:23:13.802992 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1325] Compacted 1@0 + 1@1 files to L1 => 15269756 bytes
>2018-07-11 23:23:13.836489 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.836422) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.7 rd, 73.2 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.1) out(14.6), read-write-amplify(78.8) write-amplify(37.0) OK, records in: 3644, records dropped: 508
>
>2018-07-11 23:23:13.836498 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:23:13.836443) EVENT_LOG_v1 {"time_micros": 1531351393836434, "job": 1325, "event": "compaction_finished", "compaction_time_micros": 208619, "output_level": 1, "num_output_files": 1, "total_output_size": 15269756, "num_input_records": 3644, "num_output_records": 3136, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:23:13.836637 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393836632, "job": 1325, "event": "table_file_deletion", "file_number": 11114}
>2018-07-11 23:23:13.839763 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351393839760, "job": 1325, "event": "table_file_deletion", "file_number": 11112}
>2018-07-11 23:24:11.408736 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:25:11.430790 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:26:11.461077 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:26:43.438399 7f29149e4700  1 mon.magna102@0(leader).log v22188 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22188)
>2018-07-11 23:26:57.442529 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:26:57.589845 7f29149e4700  1 mon.magna102@0(leader).log v22189 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22189)
>2018-07-11 23:27:03.481714 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:27:03.598823 7f29149e4700  1 mon.magna102@0(leader).log v22190 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22190)
>2018-07-11 23:27:11.483775 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:28:05.472450 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:28:05.646112 7f29149e4700  1 mon.magna102@0(leader).log v22191 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22191)
>2018-07-11 23:28:11.513014 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:28:21.479503 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:28:21.657062 7f29149e4700  1 mon.magna102@0(leader).log v22192 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22192)
>2018-07-11 23:28:35.485504 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 23:28:35.633269 7f29149e4700  1 mon.magna102@0(leader).log v22193 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22193)
>2018-07-11 23:28:43.525308 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:28:43.642338 7f29149e4700  1 mon.magna102@0(leader).log v22194 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22194)
>2018-07-11 23:29:11.535732 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:29:42.888724 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "status"} v 0) v1
>2018-07-11 23:29:42.888745 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/3947457421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
>2018-07-11 23:29:43.022631 7f29149e4700  1 mon.magna102@0(leader).log v22195 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22195)
>2018-07-11 23:29:51.519013 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:29:51.690270 7f29149e4700  1 mon.magna102@0(leader).log v22196 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22196)
>2018-07-11 23:30:01.523466 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:30:01.699742 7f29149e4700  1 mon.magna102@0(leader).log v22197 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22197)
>2018-07-11 23:30:07.231508 7f29191ed700  0 mon.magna102@0(leader) e1 handle_command mon_command({"prefix": "osd dump"} v 0) v1
>2018-07-11 23:30:07.231532 7f29191ed700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.107:0/606574202' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch
>2018-07-11 23:30:07.350140 7f29149e4700  1 mon.magna102@0(leader).log v22198 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22198)
>2018-07-11 23:30:11.558925 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:30:17.530244 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:30:17.676469 7f29149e4700  1 mon.magna102@0(leader).log v22199 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22199)
>2018-07-11 23:30:23.568397 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:30:23.685442 7f29149e4700  1 mon.magna102@0(leader).log v22200 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22200)
>2018-07-11 23:30:38.638317 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11116. Immutable memtables: 0.
>
>2018-07-11 23:30:38.638361 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1326] Syncing log #11113
>2018-07-11 23:30:38.654742 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.638352) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:30:38.654753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1326] Flushing memtable with next log file: 11116
>
>2018-07-11 23:30:38.654766 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351838654759, "job": 1326, "event": "flush_started", "num_memtables": 1, "num_entries": 1913, "num_deletes": 250, "memory_usage": 3632520}
>2018-07-11 23:30:38.654769 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1326] Level-0 flush table #11117: started
>2018-07-11 23:30:38.721973 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351838721959, "cf_name": "default", "job": 1326, "event": "table_file_creation", "file_number": 11117, "file_size": 1989081, "table_properties": {"data_size": 1973145, "index_size": 8196, "filter_size": 6820, "raw_key_size": 11094, "raw_average_key_size": 21, "raw_value_size": 1959992, "raw_average_value_size": 3726, "num_data_blocks": 256, "num_entries": 526, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:30:38.721984 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1326] Level-0 flush table #11117: 1989081 bytes OK
>2018-07-11 23:30:38.772034 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.738527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11117 started
>2018-07-11 23:30:38.772042 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.771997) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11117: memtable #1 done
>2018-07-11 23:30:38.772044 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.772012) EVENT_LOG_v1 {"time_micros": 1531351838772005, "job": 1326, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:30:38.772047 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.772024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:30:38.772053 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1326] Try to delete WAL files size 3595520, prev total WAL file size 3595520, number of live WAL files 2.
>
>2018-07-11 23:30:38.772917 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:30:38.772970 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:30:38.772963) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:30:38.772979 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1327] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:30:38.772985 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1326 Base level 0, inputs: [11117(1942KB)], [11115(14MB)]
>
>2018-07-11 23:30:38.772997 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351838772988, "job": 1327, "event": "compaction_started", "files_L0": [11117], "files_L1": [11115], "score": -1, "input_data_size": 17258837}
>2018-07-11 23:30:38.972748 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1327] Generated table #11118: 3370 keys, 17023044 bytes
>2018-07-11 23:30:38.972798 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351838972770, "cf_name": "default", "job": 1327, "event": "table_file_creation", "file_number": 11118, "file_size": 17023044, "table_properties": {"data_size": 16893789, "index_size": 73794, "filter_size": 54534, "raw_key_size": 68930, "raw_average_key_size": 20, "raw_value_size": 16793841, "raw_average_value_size": 4983, "num_data_blocks": 2151, "num_entries": 3370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:30:39.005892 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1327] Compacted 1@0 + 1@1 files to L1 => 17023044 bytes
>2018-07-11 23:30:39.039461 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:30:39.039391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.1 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(1.9, 14.6) out(16.2), read-write-amplify(17.2) write-amplify(8.6) OK, records in: 3662, records dropped: 292
>
>2018-07-11 23:30:39.039470 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:30:39.039412) EVENT_LOG_v1 {"time_micros": 1531351839039403, "job": 1327, "event": "compaction_finished", "compaction_time_micros": 232879, "output_level": 1, "num_output_files": 1, "total_output_size": 17023044, "num_input_records": 3662, "num_output_records": 3370, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:30:39.039910 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351839039906, "job": 1327, "event": "table_file_deletion", "file_number": 11117}
>2018-07-11 23:30:39.042816 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351839042813, "job": 1327, "event": "table_file_deletion", "file_number": 11115}
>2018-07-11 23:30:39.042871 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:31:05.700556 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11119. Immutable memtables: 0.
>
>2018-07-11 23:31:05.700597 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1328] Syncing log #11116
>2018-07-11 23:31:05.717209 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.700588) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:31:05.717220 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1328] Flushing memtable with next log file: 11119
>
>2018-07-11 23:31:05.717233 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351865717226, "job": 1328, "event": "flush_started", "num_memtables": 1, "num_entries": 354, "num_deletes": 251, "memory_usage": 235176}
>2018-07-11 23:31:05.717236 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1328] Level-0 flush table #11120: started
>2018-07-11 23:31:05.759395 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351865759383, "cf_name": "default", "job": 1328, "event": "table_file_creation", "file_number": 11120, "file_size": 218417, "table_properties": {"data_size": 215203, "index_size": 969, "filter_size": 1326, "raw_key_size": 5784, "raw_average_key_size": 20, "raw_value_size": 210673, "raw_average_value_size": 736, "num_data_blocks": 30, "num_entries": 286, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:31:05.759406 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1328] Level-0 flush table #11120: 218417 bytes OK
>2018-07-11 23:31:05.809456 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.775970) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11120 started
>2018-07-11 23:31:05.809464 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.809422) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11120: memtable #1 done
>2018-07-11 23:31:05.809466 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.809436) EVENT_LOG_v1 {"time_micros": 1531351865809430, "job": 1328, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:31:05.809468 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.809447) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:31:05.809473 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1328] Try to delete WAL files size 226105, prev total WAL file size 226105, number of live WAL files 2.
>
>2018-07-11 23:31:05.809599 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:31:05.809651 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:31:05.809644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:31:05.809660 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1329] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:31:05.809666 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1328 Base level 0, inputs: [11120(213KB)], [11118(16MB)]
>
>2018-07-11 23:31:05.809678 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351865809669, "job": 1329, "event": "compaction_started", "files_L0": [11120], "files_L1": [11118], "score": -1, "input_data_size": 17241461}
>2018-07-11 23:31:05.993533 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1329] Generated table #11121: 3148 keys, 15264273 bytes
>2018-07-11 23:31:05.993581 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351865993554, "cf_name": "default", "job": 1329, "event": "table_file_creation", "file_number": 11121, "file_size": 15264273, "table_properties": {"data_size": 15147698, "index_size": 66520, "filter_size": 49128, "raw_key_size": 64518, "raw_average_key_size": 20, "raw_value_size": 15055935, "raw_average_value_size": 4782, "num_data_blocks": 1929, "num_entries": 3148, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:31:06.026667 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1329] Compacted 1@0 + 1@1 files to L1 => 15264273 bytes
>2018-07-11 23:31:06.068550 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:31:06.068483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.5 rd, 70.4 wr, level 1, files in(1, 1) out(1) MB in(0.2, 16.2) out(14.6), read-write-amplify(148.8) write-amplify(69.9) OK, records in: 3656, records dropped: 508
>
>2018-07-11 23:31:06.068559 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:31:06.068505) EVENT_LOG_v1 {"time_micros": 1531351866068495, "job": 1329, "event": "compaction_finished", "compaction_time_micros": 216972, "output_level": 1, "num_output_files": 1, "total_output_size": 15264273, "num_input_records": 3656, "num_output_records": 3148, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:31:06.068666 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351866068662, "job": 1329, "event": "table_file_deletion", "file_number": 11120}
>2018-07-11 23:31:06.071773 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531351866071771, "job": 1329, "event": "table_file_deletion", "file_number": 11118}
>2018-07-11 23:31:11.595541 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:31:31.562806 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:31:31.738871 7f29149e4700  1 mon.magna102@0(leader).log v22201 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22201)
>2018-07-11 23:31:41.567176 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:31:41.714798 7f29149e4700  1 mon.magna102@0(leader).log v22202 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22202)
>2018-07-11 23:31:57.573825 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:31:57.724416 7f29149e4700  1 mon.magna102@0(leader).log v22203 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22203)
>2018-07-11 23:32:03.608047 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:32:03.725059 7f29149e4700  1 mon.magna102@0(leader).log v22204 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22204)
>2018-07-11 23:32:11.617148 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:33:11.647598 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:33:15.607975 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:33:15.773251 7f29149e4700  1 mon.magna102@0(leader).log v22205 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22205)
>2018-07-11 23:33:27.612982 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:33:27.758806 7f29149e4700  1 mon.magna102@0(leader).log v22206 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22206)
>2018-07-11 23:33:37.617290 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:33:37.767980 7f29149e4700  1 mon.magna102@0(leader).log v22207 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22207)
>2018-07-11 23:33:43.651583 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:33:43.768591 7f29149e4700  1 mon.magna102@0(leader).log v22208 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22208)
>2018-07-11 23:34:11.670438 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:35:01.653912 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:35:01.808850 7f29149e4700  1 mon.magna102@0(leader).log v22209 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22209)
>2018-07-11 23:35:07.656347 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:35:07.826084 7f29149e4700  1 mon.magna102@0(leader).log v22210 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22210)
>2018-07-11 23:35:11.701211 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:35:21.662661 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:35:21.835613 7f29149e4700  1 mon.magna102@0(leader).log v22211 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22211)
>2018-07-11 23:36:11.723368 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:36:35.694946 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:36:35.850449 7f29149e4700  1 mon.magna102@0(leader).log v22212 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22212)
>2018-07-11 23:36:47.700017 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:36:47.851518 7f29149e4700  1 mon.magna102@0(leader).log v22213 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22213)
>2018-07-11 23:37:01.706240 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:37:01.870565 7f29149e4700  1 mon.magna102@0(leader).log v22214 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22214)
>2018-07-11 23:37:11.746118 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:38:11.768057 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:38:21.741205 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:38:21.894260 7f29149e4700  1 mon.magna102@0(leader).log v22215 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22215)
>2018-07-11 23:38:27.743625 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:38:27.894848 7f29149e4700  1 mon.magna102@0(leader).log v22216 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22216)
>2018-07-11 23:38:45.751550 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 23:38:45.937914 7f29149e4700  1 mon.magna102@0(leader).log v22217 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22217)
>2018-07-11 23:38:51.922033 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11122. Immutable memtables: 0.
>
>2018-07-11 23:38:51.922077 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1330] Syncing log #11119
>2018-07-11 23:38:51.938577 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:51.922067) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:38:51.938588 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1330] Flushing memtable with next log file: 11122
>
>2018-07-11 23:38:51.938615 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352331938607, "job": 1330, "event": "flush_started", "num_memtables": 1, "num_entries": 2023, "num_deletes": 251, "memory_usage": 3925536}
>2018-07-11 23:38:51.938618 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1330] Level-0 flush table #11123: started
>2018-07-11 23:38:52.005764 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352332005737, "cf_name": "default", "job": 1330, "event": "table_file_creation", "file_number": 11123, "file_size": 3730003, "table_properties": {"data_size": 3699914, "index_size": 16742, "filter_size": 12422, "raw_key_size": 16070, "raw_average_key_size": 20, "raw_value_size": 3677122, "raw_average_value_size": 4732, "num_data_blocks": 503, "num_entries": 777, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:38:52.005776 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1330] Level-0 flush table #11123: 3730003 bytes OK
>2018-07-11 23:38:52.055824 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.022314) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11123 started
>2018-07-11 23:38:52.055832 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.055784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11123: memtable #1 done
>2018-07-11 23:38:52.055834 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.055798) EVENT_LOG_v1 {"time_micros": 1531352332055792, "job": 1330, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:38:52.055849 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.055812) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:38:52.055856 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1330] Try to delete WAL files size 3886482, prev total WAL file size 3886482, number of live WAL files 2.
>
>2018-07-11 23:38:52.056836 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:38:52.057034 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 23:38:52.057041 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 114358.9 total, 993.5 interval
>Cumulative writes: 141K writes, 660K keys, 141K commit groups, 1.0 writes per commit group, ingest: 1.08 GB, 0.01 MB/s
>Cumulative WAL: 141K writes, 141K syncs, 1.00 writes per sync, written: 1.08 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1062 writes, 4742 keys, 1062 commit groups, 1.0 writes per commit group, ingest: 7.76 MB, 0.01 MB/s
>Interval WAL: 1062 writes, 1062 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.56 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       666    0.078       0      0
>  L1      1/1   14.56 MB   0.0     10.5     0.9      9.6       9.6     -0.0       0.0  10.8     72.5     66.4       148       663    0.224   2384K   289K
> Sum      2/2   18.11 MB   0.0     10.5     0.9      9.6      10.5      0.9       0.0  11.8     53.8     53.8       200      1329    0.150   2384K   289K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.1     56.3     57.8         1         8    0.146     14K   1621
>Uptime(secs): 114358.9 total, 114358.9 interval
>Flush(GB): cumulative 0.891, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.50 GB write, 0.09 MB/s write, 10.50 GB read, 0.09 MB/s read, 199.8 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.56 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       666    0.078       0      0
>  L1      1/1   14.56 MB   0.0     10.5     0.9      9.6       9.6     -0.0       0.0  10.8     72.5     66.4       148       663    0.224   2384K   289K
> Sum      2/2   18.11 MB   0.0     10.5     0.9      9.6      10.5      0.9       0.0  11.8     53.8     53.8       200      1329    0.150   2384K   289K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 114358.9 total, 114358.9 interval
>Flush(GB): cumulative 0.891, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.50 GB write, 0.09 MB/s write, 10.50 GB read, 0.09 MB/s read, 199.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 23:38:52.057062 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.057055) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:38:52.057067 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1331] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:38:52.057072 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1330 Base level 0, inputs: [11123(3642KB)], [11121(14MB)]
>
>2018-07-11 23:38:52.057085 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352332057076, "job": 1331, "event": "compaction_started", "files_L0": [11123], "files_L1": [11121], "score": -1, "input_data_size": 18994276}
>2018-07-11 23:38:52.256551 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1331] Generated table #11124: 3415 keys, 17041087 bytes
>2018-07-11 23:38:52.256601 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352332256573, "cf_name": "default", "job": 1331, "event": "table_file_creation", "file_number": 11124, "file_size": 17041087, "table_properties": {"data_size": 16910009, "index_size": 75289, "filter_size": 54862, "raw_key_size": 70341, "raw_average_key_size": 20, "raw_value_size": 16808223, "raw_average_value_size": 4921, "num_data_blocks": 2180, "num_entries": 3415, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:38:52.298024 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1331] Compacted 1@0 + 1@1 files to L1 => 17041087 bytes
>2018-07-11 23:38:52.339911 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.339840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 78.8 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(3.6, 14.6) out(16.3), read-write-amplify(9.7) write-amplify(4.6) OK, records in: 3925, records dropped: 510
>
>2018-07-11 23:38:52.339920 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:52.339861) EVENT_LOG_v1 {"time_micros": 1531352332339852, "job": 1331, "event": "compaction_finished", "compaction_time_micros": 240922, "output_level": 1, "num_output_files": 1, "total_output_size": 17041087, "num_input_records": 3925, "num_output_records": 3415, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:38:52.340746 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352332340742, "job": 1331, "event": "table_file_deletion", "file_number": 11123}
>2018-07-11 23:38:52.343740 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352332343736, "job": 1331, "event": "table_file_deletion", "file_number": 11121}
>2018-07-11 23:38:53.803631 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:38:53.925324 7f29149e4700  1 mon.magna102@0(leader).log v22218 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22218)
>2018-07-11 23:38:58.867535 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11125. Immutable memtables: 0.
>
>2018-07-11 23:38:58.867600 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1332] Syncing log #11122
>2018-07-11 23:38:58.884219 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.867591) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:38:58.884232 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1332] Flushing memtable with next log file: 11125
>
>2018-07-11 23:38:58.884246 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352338884238, "job": 1332, "event": "flush_started", "num_memtables": 1, "num_entries": 284, "num_deletes": 250, "memory_usage": 86024}
>2018-07-11 23:38:58.884249 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1332] Level-0 flush table #11126: started
>2018-07-11 23:38:58.926369 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352338926356, "cf_name": "default", "job": 1332, "event": "table_file_creation", "file_number": 11126, "file_size": 78150, "table_properties": {"data_size": 76051, "index_size": 308, "filter_size": 872, "raw_key_size": 5916, "raw_average_key_size": 22, "raw_value_size": 72197, "raw_average_value_size": 269, "num_data_blocks": 10, "num_entries": 268, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:38:58.926380 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1332] Level-0 flush table #11126: 78150 bytes OK
>2018-07-11 23:38:58.976385 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.942901) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11126 started
>2018-07-11 23:38:58.976393 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.976351) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11126: memtable #1 done
>2018-07-11 23:38:58.976395 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.976366) EVENT_LOG_v1 {"time_micros": 1531352338976360, "job": 1332, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:38:58.976411 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.976376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:38:58.976416 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1332] Try to delete WAL files size 78492, prev total WAL file size 78492, number of live WAL files 2.
>
>2018-07-11 23:38:58.976527 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:38:58.976575 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:58.976569) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:38:58.976584 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1333] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:38:58.976590 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1332 Base level 0, inputs: [11126(76KB)], [11124(16MB)]
>
>2018-07-11 23:38:58.976603 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352338976593, "job": 1333, "event": "compaction_started", "files_L0": [11126], "files_L1": [11124], "score": -1, "input_data_size": 17119237}
>2018-07-11 23:38:59.168810 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1333] Generated table #11127: 3175 keys, 15400086 bytes
>2018-07-11 23:38:59.168859 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352339168832, "cf_name": "default", "job": 1333, "event": "table_file_creation", "file_number": 11127, "file_size": 15400086, "table_properties": {"data_size": 15282767, "index_size": 66873, "filter_size": 49519, "raw_key_size": 65048, "raw_average_key_size": 20, "raw_value_size": 15190376, "raw_average_value_size": 4784, "num_data_blocks": 1939, "num_entries": 3175, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:38:59.210310 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1333] Compacted 1@0 + 1@1 files to L1 => 15400086 bytes
>2018-07-11 23:38:59.243865 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:59.243796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.3 rd, 65.9 wr, level 1, files in(1, 1) out(1) MB in(0.1, 16.3) out(14.7), read-write-amplify(416.1) write-amplify(197.1) OK, records in: 3683, records dropped: 508
>
>2018-07-11 23:38:59.243873 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:38:59.243818) EVENT_LOG_v1 {"time_micros": 1531352339243808, "job": 1333, "event": "compaction_finished", "compaction_time_micros": 233692, "output_level": 1, "num_output_files": 1, "total_output_size": 15400086, "num_input_records": 3683, "num_output_records": 3175, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:38:59.243978 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352339243973, "job": 1333, "event": "table_file_deletion", "file_number": 11126}
>2018-07-11 23:38:59.247083 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352339247081, "job": 1333, "event": "table_file_deletion", "file_number": 11124}
>2018-07-11 23:38:59.247211 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:39:11.803493 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:39:59.784607 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:39:59.933439 7f29149e4700  1 mon.magna102@0(leader).log v22219 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22219)
>2018-07-11 23:40:11.789987 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:40:11.825916 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:40:11.942917 7f29149e4700  1 mon.magna102@0(leader).log v22220 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22220)
>2018-07-11 23:40:27.796754 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:40:27.952514 7f29149e4700  1 mon.magna102@0(leader).log v22221 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22221)
>2018-07-11 23:40:33.837302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:40:33.961492 7f29149e4700  1 mon.magna102@0(leader).log v22222 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22222)
>2018-07-11 23:41:11.848739 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:41:39.828236 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:41:40.026244 7f29149e4700  1 mon.magna102@0(leader).log v22223 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22223)
>2018-07-11 23:41:51.833301 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:41:51.985556 7f29149e4700  1 mon.magna102@0(leader).log v22224 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22224)
>2018-07-11 23:42:09.841079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 23:42:09.996523 7f29149e4700  1 mon.magna102@0(leader).log v22225 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22225)
>2018-07-11 23:42:11.879808 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:42:18.880166 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:42:18.980606 7f29149e4700  1 mon.magna102@0(leader).log v22226 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22226)
>2018-07-11 23:43:11.902097 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:43:25.874462 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:43:26.028420 7f29149e4700  1 mon.magna102@0(leader).log v22227 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22227)
>2018-07-11 23:43:31.877169 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:43:32.029036 7f29149e4700  1 mon.magna102@0(leader).log v22228 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22228)
>2018-07-11 23:43:49.884940 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 23:43:50.038749 7f29149e4700  1 mon.magna102@0(leader).log v22229 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22229)
>2018-07-11 23:43:55.887554 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:43:56.039348 7f29149e4700  1 mon.magna102@0(leader).log v22230 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22230)
>2018-07-11 23:44:11.932016 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:45:09.919808 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:45:10.071059 7f29149e4700  1 mon.magna102@0(leader).log v22231 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22231)
>2018-07-11 23:45:11.962713 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:45:21.925135 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:45:22.072075 7f29149e4700  1 mon.magna102@0(leader).log v22232 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22232)
>2018-07-11 23:45:35.931116 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:45:36.082775 7f29149e4700  1 mon.magna102@0(leader).log v22233 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22233)
>2018-07-11 23:46:11.985036 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:46:38.120687 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11128. Immutable memtables: 0.
>
>2018-07-11 23:46:38.120732 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1334] Syncing log #11125
>2018-07-11 23:46:38.138481 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.120722) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:46:38.138492 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1334] Flushing memtable with next log file: 11128
>
>2018-07-11 23:46:38.138506 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798138498, "job": 1334, "event": "flush_started", "num_memtables": 1, "num_entries": 1986, "num_deletes": 251, "memory_usage": 3840912}
>2018-07-11 23:46:38.138509 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1334] Level-0 flush table #11129: started
>2018-07-11 23:46:38.214048 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798214020, "cf_name": "default", "job": 1334, "event": "table_file_creation", "file_number": 11129, "file_size": 3648095, "table_properties": {"data_size": 3618603, "index_size": 16398, "filter_size": 12169, "raw_key_size": 15822, "raw_average_key_size": 20, "raw_value_size": 3596232, "raw_average_value_size": 4700, "num_data_blocks": 493, "num_entries": 765, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:46:38.214060 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1334] Level-0 flush table #11129: 3648095 bytes OK
>2018-07-11 23:46:38.264057 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.230539) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11129 started
>2018-07-11 23:46:38.264065 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.264018) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11129: memtable #1 done
>2018-07-11 23:46:38.264068 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.264033) EVENT_LOG_v1 {"time_micros": 1531352798264026, "job": 1334, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:46:38.264070 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.264047) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:46:38.264076 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1334] Try to delete WAL files size 3802905, prev total WAL file size 3802905, number of live WAL files 2.
>
>2018-07-11 23:46:38.264975 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:46:38.265031 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.265025) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:46:38.265041 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1335] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:46:38.265047 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1334 Base level 0, inputs: [11129(3562KB)], [11127(14MB)]
>
>2018-07-11 23:46:38.265060 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798265050, "job": 1335, "event": "compaction_started", "files_L0": [11129], "files_L1": [11127], "score": -1, "input_data_size": 19048181}
>2018-07-11 23:46:38.464773 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1335] Generated table #11130: 3430 keys, 17108532 bytes
>2018-07-11 23:46:38.464817 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798464790, "cf_name": "default", "job": 1335, "event": "table_file_creation", "file_number": 11130, "file_size": 17108532, "table_properties": {"data_size": 16977278, "index_size": 75301, "filter_size": 55026, "raw_key_size": 70623, "raw_average_key_size": 20, "raw_value_size": 16875271, "raw_average_value_size": 4919, "num_data_blocks": 2180, "num_entries": 3430, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:46:38.497956 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1335] Compacted 1@0 + 1@1 files to L1 => 17108532 bytes
>2018-07-11 23:46:38.531469 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.531400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 81.8 rd, 73.5 wr, level 1, files in(1, 1) out(1) MB in(3.5, 14.7) out(16.3), read-write-amplify(9.9) write-amplify(4.7) OK, records in: 3940, records dropped: 510
>
>2018-07-11 23:46:38.531477 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:46:38.531421) EVENT_LOG_v1 {"time_micros": 1531352798531412, "job": 1335, "event": "compaction_finished", "compaction_time_micros": 232880, "output_level": 1, "num_output_files": 1, "total_output_size": 17108532, "num_input_records": 3940, "num_output_records": 3430, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:46:38.532194 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798532190, "job": 1335, "event": "table_file_deletion", "file_number": 11129}
>2018-07-11 23:46:38.535206 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352798535203, "job": 1335, "event": "table_file_deletion", "file_number": 11127}
>2018-07-11 23:46:49.963530 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:46:50.132996 7f29149e4700  1 mon.magna102@0(leader).log v22234 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22234)
>2018-07-11 23:47:01.968763 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:47:02.142396 7f29149e4700  1 mon.magna102@0(leader).log v22235 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22235)
>2018-07-11 23:47:12.009591 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-11 23:47:15.974757 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:47:16.126910 7f29149e4700  1 mon.magna102@0(leader).log v22236 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22236)
>2018-07-11 23:47:19.077411 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11131. Immutable memtables: 0.
>
>2018-07-11 23:47:19.077453 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1336] Syncing log #11128
>2018-07-11 23:47:19.093931 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.077444) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:47:19.093942 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1336] Flushing memtable with next log file: 11131
>
>2018-07-11 23:47:19.093968 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839093960, "job": 1336, "event": "flush_started", "num_memtables": 1, "num_entries": 419, "num_deletes": 250, "memory_usage": 384752}
>2018-07-11 23:47:19.093971 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1336] Level-0 flush table #11132: started
>2018-07-11 23:47:19.127814 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839127801, "cf_name": "default", "job": 1336, "event": "table_file_creation", "file_number": 11132, "file_size": 267665, "table_properties": {"data_size": 264203, "index_size": 1086, "filter_size": 1457, "raw_key_size": 6444, "raw_average_key_size": 21, "raw_value_size": 259426, "raw_average_value_size": 882, "num_data_blocks": 34, "num_entries": 294, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:47:19.127823 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1336] Level-0 flush table #11132: 267665 bytes OK
>2018-07-11 23:47:19.177835 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.144343) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11132 started
>2018-07-11 23:47:19.177843 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.177800) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11132: memtable #1 done
>2018-07-11 23:47:19.177846 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.177815) EVENT_LOG_v1 {"time_micros": 1531352839177809, "job": 1336, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:47:19.177848 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.177826) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:47:19.177866 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1336] Try to delete WAL files size 374733, prev total WAL file size 374733, number of live WAL files 2.
>
>2018-07-11 23:47:19.178024 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:47:19.178072 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.178065) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:47:19.178081 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1337] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:47:19.178087 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1336 Base level 0, inputs: [11132(261KB)], [11130(16MB)]
>
>2018-07-11 23:47:19.178100 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839178091, "job": 1337, "event": "compaction_started", "files_L0": [11132], "files_L1": [11130], "score": -1, "input_data_size": 17376197}
>2018-07-11 23:47:19.363124 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1337] Generated table #11133: 3230 keys, 15753027 bytes
>2018-07-11 23:47:19.363174 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839363146, "cf_name": "default", "job": 1337, "event": "table_file_creation", "file_number": 11133, "file_size": 15753027, "table_properties": {"data_size": 15633343, "index_size": 68174, "filter_size": 50583, "raw_key_size": 66166, "raw_average_key_size": 20, "raw_value_size": 15539259, "raw_average_value_size": 4810, "num_data_blocks": 1977, "num_entries": 3230, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:47:19.404602 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1337] Compacted 1@0 + 1@1 files to L1 => 15753027 bytes
>2018-07-11 23:47:19.438124 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.438056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 76.7 rd, 69.6 wr, level 1, files in(1, 1) out(1) MB in(0.3, 16.3) out(15.0), read-write-amplify(123.8) write-amplify(58.9) OK, records in: 3724, records dropped: 494
>
>2018-07-11 23:47:19.438133 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:47:19.438078) EVENT_LOG_v1 {"time_micros": 1531352839438068, "job": 1337, "event": "compaction_finished", "compaction_time_micros": 226486, "output_level": 1, "num_output_files": 1, "total_output_size": 15753027, "num_input_records": 3724, "num_output_records": 3230, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:47:19.438281 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839438277, "job": 1337, "event": "table_file_deletion", "file_number": 11132}
>2018-07-11 23:47:19.441410 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531352839441407, "job": 1337, "event": "table_file_deletion", "file_number": 11130}
>2018-07-11 23:47:19.441456 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:48:12.033897 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:48:36.010172 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:48:36.162208 7f29149e4700  1 mon.magna102@0(leader).log v22237 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22237)
>2018-07-11 23:48:42.012727 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:48:42.229520 7f29149e4700  1 mon.magna102@0(leader).log v22238 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22238)
>2018-07-11 23:48:56.018662 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:48:56.189237 7f29149e4700  1 mon.magna102@0(leader).log v22239 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22239)
>2018-07-11 23:49:04.064612 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:49:04.198327 7f29149e4700  1 mon.magna102@0(leader).log v22240 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22240)
>2018-07-11 23:49:12.073690 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:50:12.087436 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:50:16.054312 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:50:16.204695 7f29149e4700  1 mon.magna102@0(leader).log v22241 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22241)
>2018-07-11 23:50:26.058520 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:50:26.213950 7f29149e4700  1 mon.magna102@0(leader).log v22242 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22242)
>2018-07-11 23:50:42.065489 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:50:42.224918 7f29149e4700  1 mon.magna102@0(leader).log v22243 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22243)
>2018-07-11 23:50:49.100097 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:50:49.200500 7f29149e4700  1 mon.magna102@0(leader).log v22244 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22244)
>2018-07-11 23:51:12.110412 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:52:00.099518 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:52:00.256920 7f29149e4700  1 mon.magna102@0(leader).log v22245 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22245)
>2018-07-11 23:52:06.102271 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:52:06.257438 7f29149e4700  1 mon.magna102@0(leader).log v22246 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22246)
>2018-07-11 23:52:12.149422 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-11 23:52:22.109132 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:52:22.283776 7f29149e4700  1 mon.magna102@0(leader).log v22247 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22247)
>2018-07-11 23:52:29.150667 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:52:29.276044 7f29149e4700  1 mon.magna102@0(leader).log v22248 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22248)
>2018-07-11 23:53:12.164345 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:53:40.143365 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:53:40.300045 7f29149e4700  1 mon.magna102@0(leader).log v22249 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22249)
>2018-07-11 23:53:52.148539 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:53:52.301041 7f29149e4700  1 mon.magna102@0(leader).log v22250 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22250)
>2018-07-11 23:54:06.154585 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:54:06.328399 7f29149e4700  1 mon.magna102@0(leader).log v22251 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22251)
>2018-07-11 23:54:12.203784 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:54:14.195570 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:54:14.320942 7f29149e4700  1 mon.magna102@0(leader).log v22252 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22252)
>2018-07-11 23:54:18.288283 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11134. Immutable memtables: 0.
>
>2018-07-11 23:54:18.288326 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1338] Syncing log #11131
>2018-07-11 23:54:18.304817 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.288317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:54:18.304828 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1338] Flushing memtable with next log file: 11134
>
>2018-07-11 23:54:18.304854 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258304846, "job": 1338, "event": "flush_started", "num_memtables": 1, "num_entries": 1854, "num_deletes": 251, "memory_usage": 3563264}
>2018-07-11 23:54:18.304858 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1338] Level-0 flush table #11135: started
>2018-07-11 23:54:18.372033 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258372018, "cf_name": "default", "job": 1338, "event": "table_file_creation", "file_number": 11135, "file_size": 3386558, "table_properties": {"data_size": 3359196, "index_size": 15112, "filter_size": 11326, "raw_key_size": 15043, "raw_average_key_size": 20, "raw_value_size": 3338244, "raw_average_value_size": 4585, "num_data_blocks": 455, "num_entries": 728, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-11 23:54:18.372057 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1338] Level-0 flush table #11135: 3386558 bytes OK
>2018-07-11 23:54:18.422061 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.388556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11135 started
>2018-07-11 23:54:18.422069 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.422022) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11135: memtable #1 done
>2018-07-11 23:54:18.422071 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.422037) EVENT_LOG_v1 {"time_micros": 1531353258422031, "job": 1338, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:54:18.422086 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.422050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:54:18.422093 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1338] Try to delete WAL files size 3527551, prev total WAL file size 3527551, number of live WAL files 2.
>
>2018-07-11 23:54:18.422971 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:54:18.423167 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-11 23:54:18.423174 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 115285.3 total, 926.4 interval
>Cumulative writes: 142K writes, 664K keys, 142K commit groups, 1.0 writes per commit group, ingest: 1.08 GB, 0.01 MB/s
>Cumulative WAL: 142K writes, 142K syncs, 1.00 writes per sync, written: 1.08 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4543 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.42 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.23 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       670    0.077       0      0
>  L1      1/1   15.02 MB   0.0     10.6     0.9      9.7       9.7     -0.0       0.0  10.8     72.6     66.4       149       667    0.224   2399K   291K
> Sum      2/2   18.25 MB   0.0     10.6     0.9      9.7      10.6      0.9       0.0  11.8     53.8     53.8       201      1337    0.150   2399K   291K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0   9.8     56.8     56.9         1         8    0.152     15K   2022
>Uptime(secs): 115285.3 total, 115285.3 interval
>Flush(GB): cumulative 0.898, interval 0.007
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.57 GB write, 0.09 MB/s write, 10.57 GB read, 0.09 MB/s read, 201.0 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    3.23 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       670    0.077       0      0
>  L1      1/1   15.02 MB   0.0     10.6     0.9      9.7       9.7     -0.0       0.0  10.8     72.6     66.4       149       667    0.224   2399K   291K
> Sum      2/2   18.25 MB   0.0     10.6     0.9      9.7      10.6      0.9       0.0  11.8     53.8     53.8       201      1337    0.150   2399K   291K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 115285.3 total, 115285.3 interval
>Flush(GB): cumulative 0.898, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.57 GB write, 0.09 MB/s write, 10.57 GB read, 0.09 MB/s read, 201.0 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-11 23:54:18.423203 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.423196) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-11 23:54:18.423208 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1339] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:54:18.423213 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1338 Base level 0, inputs: [11135(3307KB)], [11133(15MB)]
>
>2018-07-11 23:54:18.423226 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258423217, "job": 1339, "event": "compaction_started", "files_L0": [11135], "files_L1": [11133], "score": -1, "input_data_size": 19139585}
>2018-07-11 23:54:18.622776 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1339] Generated table #11136: 3448 keys, 17169927 bytes
>2018-07-11 23:54:18.622820 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258622793, "cf_name": "default", "job": 1339, "event": "table_file_creation", "file_number": 11136, "file_size": 17169927, "table_properties": {"data_size": 17038500, "index_size": 75314, "filter_size": 55186, "raw_key_size": 70962, "raw_average_key_size": 20, "raw_value_size": 16936229, "raw_average_value_size": 4911, "num_data_blocks": 2180, "num_entries": 3448, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:54:18.664249 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1339] Compacted 1@0 + 1@1 files to L1 => 17169927 bytes
>2018-07-11 23:54:18.697824 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.697756) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.4 rd, 71.2 wr, level 1, files in(1, 1) out(1) MB in(3.2, 15.0) out(16.4), read-write-amplify(10.7) write-amplify(5.1) OK, records in: 3958, records dropped: 510
>
>2018-07-11 23:54:18.697833 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:54:18.697778) EVENT_LOG_v1 {"time_micros": 1531353258697769, "job": 1339, "event": "compaction_finished", "compaction_time_micros": 241009, "output_level": 1, "num_output_files": 1, "total_output_size": 17169927, "num_input_records": 3958, "num_output_records": 3448, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:54:18.698579 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258698575, "job": 1339, "event": "table_file_deletion", "file_number": 11135}
>2018-07-11 23:54:18.701650 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353258701647, "job": 1339, "event": "table_file_deletion", "file_number": 11133}
>2018-07-11 23:55:12.219800 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-11 23:55:20.187268 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:55:20.337402 7f29149e4700  1 mon.magna102@0(leader).log v22253 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22253)
>2018-07-11 23:55:32.192481 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:55:32.346687 7f29149e4700  1 mon.magna102@0(leader).log v22254 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22254)
>2018-07-11 23:55:39.289123 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11137. Immutable memtables: 0.
>
>2018-07-11 23:55:39.289166 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1340] Syncing log #11134
>2018-07-11 23:55:39.305677 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.289157) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-11 23:55:39.305689 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1340] Flushing memtable with next log file: 11137
>
>2018-07-11 23:55:39.305702 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339305695, "job": 1340, "event": "flush_started", "num_memtables": 1, "num_entries": 551, "num_deletes": 250, "memory_usage": 674816}
>2018-07-11 23:55:39.305704 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1340] Level-0 flush table #11138: started
>2018-07-11 23:55:39.347877 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339347864, "cf_name": "default", "job": 1340, "event": "table_file_creation", "file_number": 11138, "file_size": 403583, "table_properties": {"data_size": 399127, "index_size": 1657, "filter_size": 1880, "raw_key_size": 6783, "raw_average_key_size": 21, "raw_value_size": 393698, "raw_average_value_size": 1265, "num_data_blocks": 52, "num_entries": 311, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-11 23:55:39.347887 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1340] Level-0 flush table #11138: 403583 bytes OK
>2018-07-11 23:55:39.397934 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.364443) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11138 started
>2018-07-11 23:55:39.397942 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.397899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11138: memtable #1 done
>2018-07-11 23:55:39.397945 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.397913) EVENT_LOG_v1 {"time_micros": 1531353339397907, "job": 1340, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-11 23:55:39.397947 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.397925) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-11 23:55:39.397953 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1340] Try to delete WAL files size 662719, prev total WAL file size 662719, number of live WAL files 2.
>
>2018-07-11 23:55:39.398158 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:55:39.398218 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.398211) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-11 23:55:39.398228 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1341] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-11 23:55:39.398234 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1340 Base level 0, inputs: [11138(394KB)], [11136(16MB)]
>
>2018-07-11 23:55:39.398246 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339398237, "job": 1341, "event": "compaction_started", "files_L0": [11138], "files_L1": [11136], "score": -1, "input_data_size": 17573510}
>2018-07-11 23:55:39.590319 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1341] Generated table #11139: 3285 keys, 16087500 bytes
>2018-07-11 23:55:39.590355 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339590339, "cf_name": "default", "job": 1341, "event": "table_file_creation", "file_number": 11139, "file_size": 16087500, "table_properties": {"data_size": 15965489, "index_size": 69474, "filter_size": 51610, "raw_key_size": 67284, "raw_average_key_size": 20, "raw_value_size": 15869711, "raw_average_value_size": 4830, "num_data_blocks": 2015, "num_entries": 3285, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-11 23:55:39.631854 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1341] Compacted 1@0 + 1@1 files to L1 => 16087500 bytes
>2018-07-11 23:55:39.665359 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.665293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.2 rd, 68.9 wr, level 1, files in(1, 1) out(1) MB in(0.4, 16.4) out(15.3), read-write-amplify(83.4) write-amplify(39.9) OK, records in: 3759, records dropped: 474
>
>2018-07-11 23:55:39.665368 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/11-23:55:39.665314) EVENT_LOG_v1 {"time_micros": 1531353339665305, "job": 1341, "event": "compaction_finished", "compaction_time_micros": 233589, "output_level": 1, "num_output_files": 1, "total_output_size": 16087500, "num_input_records": 3759, "num_output_records": 3285, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-11 23:55:39.665504 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339665500, "job": 1341, "event": "table_file_deletion", "file_number": 11138}
>2018-07-11 23:55:39.668612 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353339668609, "job": 1341, "event": "table_file_deletion", "file_number": 11136}
>2018-07-11 23:55:39.668668 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-11 23:55:46.198701 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:55:46.351235 7f29149e4700  1 mon.magna102@0(leader).log v22255 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22255)
>2018-07-11 23:55:54.256071 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:55:54.376969 7f29149e4700  1 mon.magna102@0(leader).log v22256 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22256)
>2018-07-11 23:56:12.244702 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:57:06.233760 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:57:06.408275 7f29149e4700  1 mon.magna102@0(leader).log v22257 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22257)
>2018-07-11 23:57:12.275061 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:57:22.240779 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:57:22.409642 7f29149e4700  1 mon.magna102@0(leader).log v22258 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22258)
>2018-07-11 23:57:30.244130 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
>2018-07-11 23:57:30.402014 7f29149e4700  1 mon.magna102@0(leader).log v22259 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22259)
>2018-07-11 23:57:39.285163 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:57:39.377744 7f29149e4700  1 mon.magna102@0(leader).log v22260 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22260)
>2018-07-11 23:58:12.289447 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:58:46.278421 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:58:46.433894 7f29149e4700  1 mon.magna102@0(leader).log v22261 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22261)
>2018-07-11 23:59:02.285522 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:59:02.460256 7f29149e4700  1 mon.magna102@0(leader).log v22262 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22262)
>2018-07-11 23:59:12.289634 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-11 23:59:16.292000 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 3,4 (REQUEST_SLOW)
>2018-07-11 23:59:16.437620 7f29149e4700  1 mon.magna102@0(leader).log v22263 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22263)
>2018-07-11 23:59:40.302594 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)
>2018-07-11 23:59:40.456083 7f29149e4700  1 mon.magna102@0(leader).log v22264 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22264)
>2018-07-11 23:59:43.439659 7f29149e4700  1 mon.magna102@0(leader).log v22265 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22265)
>2018-07-11 23:59:45.573318 7f29149e4700  1 mon.magna102@0(leader).log v22266 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22266)
>2018-07-11 23:59:46.573346 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 4)
>2018-07-11 23:59:46.732043 7f29149e4700  1 mon.magna102@0(leader).log v22267 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22267)
>2018-07-12 00:00:07.450536 7f29149e4700  1 mon.magna102@0(leader).log v22268 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22268)
>2018-07-12 00:00:12.289822 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 00:01:12.290005 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 00:02:04.530113 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11140. Immutable memtables: 0.
>
>2018-07-12 00:02:04.530248 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1342] Syncing log #11137
>2018-07-12 00:02:04.546414 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.530238) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:02:04.546425 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1342] Flushing memtable with next log file: 11140
>
>2018-07-12 00:02:04.546452 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724546444, "job": 1342, "event": "flush_started", "num_memtables": 1, "num_entries": 1719, "num_deletes": 251, "memory_usage": 3232096}
>2018-07-12 00:02:04.546455 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1342] Level-0 flush table #11141: started
>2018-07-12 00:02:04.613625 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724613610, "cf_name": "default", "job": 1342, "event": "table_file_creation", "file_number": 11141, "file_size": 3078209, "table_properties": {"data_size": 3053112, "index_size": 13801, "filter_size": 10372, "raw_key_size": 14207, "raw_average_key_size": 20, "raw_value_size": 3033662, "raw_average_value_size": 4409, "num_data_blocks": 416, "num_entries": 688, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:02:04.613636 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1342] Level-0 flush table #11141: 3078209 bytes OK
>2018-07-12 00:02:04.671973 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.630155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11141 started
>2018-07-12 00:02:04.671981 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.671936) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11141: memtable #1 done
>2018-07-12 00:02:04.671984 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.671950) EVENT_LOG_v1 {"time_micros": 1531353724671944, "job": 1342, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:02:04.671998 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.671963) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:02:04.672005 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1342] Try to delete WAL files size 3198287, prev total WAL file size 3198287, number of live WAL files 2.
>
>2018-07-12 00:02:04.672828 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:02:04.672891 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.672883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:02:04.672897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1343] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:02:04.672905 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1342 Base level 0, inputs: [11141(3006KB)], [11139(15MB)]
>
>2018-07-12 00:02:04.672919 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724672909, "job": 1343, "event": "compaction_started", "files_L0": [11141], "files_L1": [11139], "score": -1, "input_data_size": 19165709}
>2018-07-12 00:02:04.872687 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1343] Generated table #11142: 3463 keys, 17196253 bytes
>2018-07-12 00:02:04.872730 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724872704, "cf_name": "default", "job": 1343, "event": "table_file_creation", "file_number": 11142, "file_size": 17196253, "table_properties": {"data_size": 17064760, "index_size": 75297, "filter_size": 55269, "raw_key_size": 71244, "raw_average_key_size": 20, "raw_value_size": 16962291, "raw_average_value_size": 4898, "num_data_blocks": 2179, "num_entries": 3463, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:02:04.905882 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1343] Compacted 1@0 + 1@1 files to L1 => 17196253 bytes
>2018-07-12 00:02:04.939410 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.939336) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 82.3 rd, 73.8 wr, level 1, files in(1, 1) out(1) MB in(2.9, 15.3) out(16.4), read-write-amplify(11.8) write-amplify(5.6) OK, records in: 3973, records dropped: 510
>
>2018-07-12 00:02:04.939419 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:02:04.939357) EVENT_LOG_v1 {"time_micros": 1531353724939347, "job": 1343, "event": "compaction_finished", "compaction_time_micros": 232945, "output_level": 1, "num_output_files": 1, "total_output_size": 17196253, "num_input_records": 3973, "num_output_records": 3463, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:02:04.940069 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724940065, "job": 1343, "event": "table_file_deletion", "file_number": 11141}
>2018-07-12 00:02:04.943206 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353724943203, "job": 1343, "event": "table_file_deletion", "file_number": 11139}
>2018-07-12 00:02:12.290192 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:03:12.290361 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:03:59.542242 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11143. Immutable memtables: 0.
>
>2018-07-12 00:03:59.542285 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1344] Syncing log #11140
>2018-07-12 00:03:59.558772 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.542275) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:03:59.558784 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1344] Flushing memtable with next log file: 11143
>
>2018-07-12 00:03:59.558810 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839558802, "job": 1344, "event": "flush_started", "num_memtables": 1, "num_entries": 654, "num_deletes": 250, "memory_usage": 869832}
>2018-07-12 00:03:59.558813 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1344] Level-0 flush table #11144: started
>2018-07-12 00:03:59.600963 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839600948, "cf_name": "default", "job": 1344, "event": "table_file_creation", "file_number": 11144, "file_size": 480548, "table_properties": {"data_size": 475377, "index_size": 2101, "filter_size": 2151, "raw_key_size": 6950, "raw_average_key_size": 21, "raw_value_size": 469527, "raw_average_value_size": 1467, "num_data_blocks": 66, "num_entries": 320, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:03:59.600973 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1344] Level-0 flush table #11144: 480548 bytes OK
>2018-07-12 00:03:59.650998 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.617500) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11144 started
>2018-07-12 00:03:59.651006 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.650962) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11144: memtable #1 done
>2018-07-12 00:03:59.651009 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.650977) EVENT_LOG_v1 {"time_micros": 1531353839650971, "job": 1344, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:03:59.651023 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.650988) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:03:59.651030 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1344] Try to delete WAL files size 855164, prev total WAL file size 855164, number of live WAL files 2.
>
>2018-07-12 00:03:59.651295 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:03:59.651344 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.651337) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:03:59.651353 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1345] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:03:59.651359 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1344 Base level 0, inputs: [11144(469KB)], [11142(16MB)]
>
>2018-07-12 00:03:59.651373 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839651363, "job": 1345, "event": "compaction_started", "files_L0": [11144], "files_L1": [11142], "score": -1, "input_data_size": 17676801}
>2018-07-12 00:03:59.851700 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1345] Generated table #11145: 3328 keys, 16312664 bytes
>2018-07-12 00:03:59.851744 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839851716, "cf_name": "default", "job": 1345, "event": "table_file_creation", "file_number": 11145, "file_size": 16312664, "table_properties": {"data_size": 16188925, "index_size": 70470, "filter_size": 52342, "raw_key_size": 68158, "raw_average_key_size": 20, "raw_value_size": 16091837, "raw_average_value_size": 4835, "num_data_blocks": 2044, "num_entries": 3328, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:03:59.893225 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1345] Compacted 1@0 + 1@1 files to L1 => 16312664 bytes
>2018-07-12 00:03:59.926756 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.926689) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.1 rd, 67.5 wr, level 1, files in(1, 1) out(1) MB in(0.5, 16.4) out(15.6), read-write-amplify(70.7) write-amplify(33.9) OK, records in: 3783, records dropped: 455
>
>2018-07-12 00:03:59.926765 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:03:59.926711) EVENT_LOG_v1 {"time_micros": 1531353839926701, "job": 1345, "event": "compaction_finished", "compaction_time_micros": 241836, "output_level": 1, "num_output_files": 1, "total_output_size": 16312664, "num_input_records": 3783, "num_output_records": 3328, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:03:59.926931 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839926926, "job": 1345, "event": "table_file_deletion", "file_number": 11144}
>2018-07-12 00:03:59.930064 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531353839930062, "job": 1345, "event": "table_file_deletion", "file_number": 11142}
>2018-07-12 00:03:59.930196 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:04:12.290535 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:05:12.290709 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:06:12.290897 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:07:12.291069 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:08:12.291255 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:09:12.291427 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:10:12.291616 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:10:22.729084 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11146. Immutable memtables: 0.
>
>2018-07-12 00:10:22.729128 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1346] Syncing log #11143
>2018-07-12 00:10:22.745603 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.729118) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:10:22.745614 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1346] Flushing memtable with next log file: 11146
>
>2018-07-12 00:10:22.745641 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354222745633, "job": 1346, "event": "flush_started", "num_memtables": 1, "num_entries": 1600, "num_deletes": 251, "memory_usage": 2880304}
>2018-07-12 00:10:22.745645 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1346] Level-0 flush table #11147: started
>2018-07-12 00:10:22.804480 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354222804466, "cf_name": "default", "job": 1346, "event": "table_file_creation", "file_number": 11147, "file_size": 2747031, "table_properties": {"data_size": 2723918, "index_size": 12763, "filter_size": 9426, "raw_key_size": 13260, "raw_average_key_size": 20, "raw_value_size": 2705862, "raw_average_value_size": 4214, "num_data_blocks": 386, "num_entries": 642, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:10:22.804492 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1346] Level-0 flush table #11147: 2747031 bytes OK
>2018-07-12 00:10:22.854484 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.820986) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11147 started
>2018-07-12 00:10:22.854492 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.854446) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11147: memtable #1 done
>2018-07-12 00:10:22.854494 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.854461) EVENT_LOG_v1 {"time_micros": 1531354222854454, "job": 1346, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:10:22.854508 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.854473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:10:22.854515 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1346] Try to delete WAL files size 2847969, prev total WAL file size 2847969, number of live WAL files 2.
>
>2018-07-12 00:10:22.855266 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:10:22.855450 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 00:10:22.855456 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 116249.7 total, 964.4 interval
>Cumulative writes: 143K writes, 669K keys, 143K commit groups, 1.0 writes per commit group, ingest: 1.09 GB, 0.01 MB/s
>Cumulative WAL: 143K writes, 143K syncs, 1.00 writes per sync, written: 1.09 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 1004 writes, 4524 keys, 1004 commit groups, 1.0 writes per commit group, ingest: 7.21 MB, 0.01 MB/s
>Interval WAL: 1004 writes, 1004 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.62 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       674    0.077       0      0
>  L1      1/1   15.56 MB   0.0     10.6     0.9      9.7       9.7      0.0       0.0  10.8     72.6     66.4       150       671    0.224   2415K   293K
> Sum      2/2   18.18 MB   0.0     10.6     0.9      9.7      10.6      0.9       0.0  11.8     53.8     53.8       202      1345    0.150   2415K   293K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  11.0     57.2     57.2         1         8    0.153     15K   1949
>Uptime(secs): 116249.7 total, 116249.7 interval
>Flush(GB): cumulative 0.904, interval 0.006
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.64 GB write, 0.09 MB/s write, 10.63 GB read, 0.09 MB/s read, 202.3 seconds
>Interval compaction: 0.07 GB write, 0.00 MB/s write, 0.07 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.62 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        52       674    0.077       0      0
>  L1      1/1   15.56 MB   0.0     10.6     0.9      9.7       9.7      0.0       0.0  10.8     72.6     66.4       150       671    0.224   2415K   293K
> Sum      2/2   18.18 MB   0.0     10.6     0.9      9.7      10.6      0.9       0.0  11.8     53.8     53.8       202      1345    0.150   2415K   293K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 116249.7 total, 116249.7 interval
>Flush(GB): cumulative 0.904, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.64 GB write, 0.09 MB/s write, 10.63 GB read, 0.09 MB/s read, 202.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 00:10:22.855480 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:10:22.855473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:10:22.855485 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1347] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:10:22.855490 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1346 Base level 0, inputs: [11147(2682KB)], [11145(15MB)]
>
>2018-07-12 00:10:22.855503 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354222855494, "job": 1347, "event": "compaction_started", "files_L0": [11147], "files_L1": [11145], "score": -1, "input_data_size": 19059695}
>2018-07-12 00:10:23.063531 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1347] Generated table #11148: 3462 keys, 17079813 bytes
>2018-07-12 00:10:23.063576 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354223063548, "cf_name": "default", "job": 1347, "event": "table_file_creation", "file_number": 11148, "file_size": 17079813, "table_properties": {"data_size": 16948591, "index_size": 75263, "filter_size": 55032, "raw_key_size": 71222, "raw_average_key_size": 20, "raw_value_size": 16846161, "raw_average_value_size": 4866, "num_data_blocks": 2178, "num_entries": 3462, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:10:23.096689 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1347] Compacted 1@0 + 1@1 files to L1 => 17079813 bytes
>2018-07-12 00:10:23.130232 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:10:23.130160) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 79.0 rd, 70.8 wr, level 1, files in(1, 1) out(1) MB in(2.6, 15.6) out(16.3), read-write-amplify(13.2) write-amplify(6.2) OK, records in: 3970, records dropped: 508
>
>2018-07-12 00:10:23.130241 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:10:23.130188) EVENT_LOG_v1 {"time_micros": 1531354223130172, "job": 1347, "event": "compaction_finished", "compaction_time_micros": 241170, "output_level": 1, "num_output_files": 1, "total_output_size": 17079813, "num_input_records": 3970, "num_output_records": 3462, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:10:23.130834 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354223130830, "job": 1347, "event": "table_file_deletion", "file_number": 11147}
>2018-07-12 00:10:23.134009 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354223134006, "job": 1347, "event": "table_file_deletion", "file_number": 11145}
>2018-07-12 00:11:12.291800 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:12:12.291979 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:12:19.739990 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11149. Immutable memtables: 0.
>
>2018-07-12 00:12:19.740033 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1348] Syncing log #11146
>2018-07-12 00:12:19.756506 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.740023) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:12:19.756517 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1348] Flushing memtable with next log file: 11149
>
>2018-07-12 00:12:19.756544 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354339756535, "job": 1348, "event": "flush_started", "num_memtables": 1, "num_entries": 661, "num_deletes": 250, "memory_usage": 884312}
>2018-07-12 00:12:19.756547 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1348] Level-0 flush table #11150: started
>2018-07-12 00:12:19.798738 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354339798724, "cf_name": "default", "job": 1348, "event": "table_file_creation", "file_number": 11150, "file_size": 488004, "table_properties": {"data_size": 482779, "index_size": 2134, "filter_size": 2172, "raw_key_size": 6970, "raw_average_key_size": 21, "raw_value_size": 476890, "raw_average_value_size": 1485, "num_data_blocks": 67, "num_entries": 321, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:12:19.798748 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1348] Level-0 flush table #11150: 488004 bytes OK
>2018-07-12 00:12:19.848751 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.815270) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11150 started
>2018-07-12 00:12:19.848759 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.848715) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11150: memtable #1 done
>2018-07-12 00:12:19.848762 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.848730) EVENT_LOG_v1 {"time_micros": 1531354339848723, "job": 1348, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:12:19.848776 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.848741) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:12:19.848783 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1348] Try to delete WAL files size 869933, prev total WAL file size 869933, number of live WAL files 2.
>
>2018-07-12 00:12:19.849069 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:12:19.849119 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:12:19.849112) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:12:19.849129 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1349] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:12:19.849135 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1348 Base level 0, inputs: [11150(476KB)], [11148(16MB)]
>
>2018-07-12 00:12:19.849148 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354339849138, "job": 1349, "event": "compaction_started", "files_L0": [11150], "files_L1": [11148], "score": -1, "input_data_size": 17567817}
>2018-07-12 00:12:20.041118 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1349] Generated table #11151: 3329 keys, 16220172 bytes
>2018-07-12 00:12:20.041167 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354340041139, "cf_name": "default", "job": 1349, "event": "table_file_creation", "file_number": 11151, "file_size": 16220172, "table_properties": {"data_size": 16096569, "index_size": 70505, "filter_size": 52171, "raw_key_size": 68178, "raw_average_key_size": 20, "raw_value_size": 15999444, "raw_average_value_size": 4806, "num_data_blocks": 2045, "num_entries": 3329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:12:20.082617 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1349] Compacted 1@0 + 1@1 files to L1 => 16220172 bytes
>2018-07-12 00:12:20.117377 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:12:20.117304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.3 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(0.5, 16.3) out(15.5), read-write-amplify(69.2) write-amplify(33.2) OK, records in: 3783, records dropped: 454
>
>2018-07-12 00:12:20.117386 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:12:20.117325) EVENT_LOG_v1 {"time_micros": 1531354340117316, "job": 1349, "event": "compaction_finished", "compaction_time_micros": 233452, "output_level": 1, "num_output_files": 1, "total_output_size": 16220172, "num_input_records": 3783, "num_output_records": 3329, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:12:20.117569 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354340117564, "job": 1349, "event": "table_file_deletion", "file_number": 11150}
>2018-07-12 00:12:20.120702 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354340120699, "job": 1349, "event": "table_file_deletion", "file_number": 11148}
>2018-07-12 00:12:20.120820 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:12:40.786124 7f29149e4700  1 mon.magna102@0(leader).log v22269 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22269)
>2018-07-12 00:13:12.292155 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:14:12.292353 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:15:12.292537 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:16:12.292709 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:17:12.292894 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:18:12.293070 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 00:18:23.930320 7f29149e4700  1 mon.magna102@0(leader).log v22270 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22270)
>2018-07-12 00:18:24.897835 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11152. Immutable memtables: 0.
>
>2018-07-12 00:18:24.897904 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1350] Syncing log #11149
>2018-07-12 00:18:24.913928 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:24.897893) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:18:24.913939 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1350] Flushing memtable with next log file: 11152
>
>2018-07-12 00:18:24.913966 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354704913958, "job": 1350, "event": "flush_started", "num_memtables": 1, "num_entries": 1795, "num_deletes": 500, "memory_usage": 2789264}
>2018-07-12 00:18:24.913969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1350] Level-0 flush table #11153: started
>2018-07-12 00:18:24.982560 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354704982532, "cf_name": "default", "job": 1350, "event": "table_file_creation", "file_number": 11153, "file_size": 2655634, "table_properties": {"data_size": 2632724, "index_size": 12250, "filter_size": 9735, "raw_key_size": 18242, "raw_average_key_size": 20, "raw_value_size": 2611915, "raw_average_value_size": 2971, "num_data_blocks": 371, "num_entries": 879, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "500", "kMergeOperands": "0"}}
>2018-07-12 00:18:24.982572 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1350] Level-0 flush table #11153: 2655634 bytes OK
>2018-07-12 00:18:25.074258 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.022922) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11153 started
>2018-07-12 00:18:25.074266 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.074207) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11153: memtable #1 done
>2018-07-12 00:18:25.074269 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.074222) EVENT_LOG_v1 {"time_micros": 1531354705074215, "job": 1350, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:18:25.074271 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.074247) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:18:25.074277 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1350] Try to delete WAL files size 2750930, prev total WAL file size 2765699, number of live WAL files 2.
>
>2018-07-12 00:18:25.075000 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:18:25.075062 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.075055) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-12 00:18:25.075069 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1351] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:18:25.075075 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1350 Base level 0, inputs: [11153(2593KB)], [11151(15MB)]
>
>2018-07-12 00:18:25.075089 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705075078, "job": 1351, "event": "compaction_started", "files_L0": [11153], "files_L1": [11151], "score": -1, "input_data_size": 18875806}
>2018-07-12 00:18:25.266608 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1351] Generated table #11154: 3200 keys, 16491507 bytes
>2018-07-12 00:18:25.266657 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705266630, "cf_name": "default", "job": 1351, "event": "table_file_creation", "file_number": 11154, "file_size": 16491507, "table_properties": {"data_size": 16363797, "index_size": 73658, "filter_size": 53125, "raw_key_size": 65714, "raw_average_key_size": 20, "raw_value_size": 16266308, "raw_average_value_size": 5083, "num_data_blocks": 2152, "num_entries": 3200, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:18:25.299784 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1351] Compacted 1@0 + 1@1 files to L1 => 16491507 bytes
>2018-07-12 00:18:25.333294 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.333228) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 84.0 rd, 73.4 wr, level 1, files in(1, 1) out(1) MB in(2.5, 15.5) out(15.7), read-write-amplify(13.3) write-amplify(6.2) OK, records in: 4208, records dropped: 1008
>
>2018-07-12 00:18:25.333303 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.333249) EVENT_LOG_v1 {"time_micros": 1531354705333240, "job": 1351, "event": "compaction_finished", "compaction_time_micros": 224678, "output_level": 1, "num_output_files": 1, "total_output_size": 16491507, "num_input_records": 4208, "num_output_records": 3200, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:18:25.333878 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705333874, "job": 1351, "event": "table_file_deletion", "file_number": 11153}
>2018-07-12 00:18:25.337046 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705337044, "job": 1351, "event": "table_file_deletion", "file_number": 11151}
>2018-07-12 00:18:25.337394 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11155. Immutable memtables: 0.
>
>2018-07-12 00:18:25.337433 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1352] Syncing log #11152
>2018-07-12 00:18:25.350235 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.337425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:18:25.350246 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1352] Flushing memtable with next log file: 11155
>
>2018-07-12 00:18:25.350273 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705350265, "job": 1352, "event": "flush_started", "num_memtables": 1, "num_entries": 7, "num_deletes": 0, "memory_usage": 17120}
>2018-07-12 00:18:25.350276 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1352] Level-0 flush table #11156: started
>2018-07-12 00:18:25.392376 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705392363, "cf_name": "default", "job": 1352, "event": "table_file_creation", "file_number": 11156, "file_size": 15875, "table_properties": {"data_size": 14820, "index_size": 81, "filter_size": 64, "raw_key_size": 169, "raw_average_key_size": 24, "raw_value_size": 14608, "raw_average_value_size": 2086, "num_data_blocks": 3, "num_entries": 7, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:18:25.392386 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1352] Level-0 flush table #11156: 15875 bytes OK
>2018-07-12 00:18:25.442426 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.408914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11156 started
>2018-07-12 00:18:25.442434 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.442392) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11156: memtable #1 done
>2018-07-12 00:18:25.442436 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.442406) EVENT_LOG_v1 {"time_micros": 1531354705442400, "job": 1352, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:18:25.442452 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.442418) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:18:25.442457 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1352] Try to delete WAL files size 14769, prev total WAL file size 14769, number of live WAL files 2.
>
>2018-07-12 00:18:25.442555 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:18:25.442601 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.442595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'logm .. 'logm; will stop at (end)
>
>2018-07-12 00:18:25.442610 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1353] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:18:25.442616 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1352 Base level 0, inputs: [11156(15KB)], [11154(15MB)]
>
>2018-07-12 00:18:25.442629 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705442619, "job": 1353, "event": "compaction_started", "files_L0": [11156], "files_L1": [11154], "score": -1, "input_data_size": 16507382}
>2018-07-12 00:18:25.634830 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1353] Generated table #11157: 3202 keys, 16505742 bytes
>2018-07-12 00:18:25.634874 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705634847, "cf_name": "default", "job": 1353, "event": "table_file_creation", "file_number": 11157, "file_size": 16505742, "table_properties": {"data_size": 16377918, "index_size": 73726, "filter_size": 53171, "raw_key_size": 65756, "raw_average_key_size": 20, "raw_value_size": 16280353, "raw_average_value_size": 5084, "num_data_blocks": 2154, "num_entries": 3202, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:18:25.668025 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1353] Compacted 1@0 + 1@1 files to L1 => 16505742 bytes
>2018-07-12 00:18:25.701529 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.701458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.2 rd, 73.2 wr, level 1, files in(1, 1) out(1) MB in(0.0, 15.7) out(15.7), read-write-amplify(2079.6) write-amplify(1039.7) OK, records in: 3207, records dropped: 5
>
>2018-07-12 00:18:25.701537 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:25.701480) EVENT_LOG_v1 {"time_micros": 1531354705701470, "job": 1353, "event": "compaction_finished", "compaction_time_micros": 225382, "output_level": 1, "num_output_files": 1, "total_output_size": 16505742, "num_input_records": 3207, "num_output_records": 3202, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:18:25.701632 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705701627, "job": 1353, "event": "table_file_deletion", "file_number": 11156}
>2018-07-12 00:18:25.704635 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354705704632, "job": 1353, "event": "table_file_deletion", "file_number": 11154}
>2018-07-12 00:18:34.952985 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11158. Immutable memtables: 0.
>
>2018-07-12 00:18:34.953027 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1354] Syncing log #11155
>2018-07-12 00:18:34.969568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:34.953018) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:18:34.969580 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1354] Flushing memtable with next log file: 11158
>
>2018-07-12 00:18:34.969605 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354714969597, "job": 1354, "event": "flush_started", "num_memtables": 1, "num_entries": 291, "num_deletes": 251, "memory_usage": 94400}
>2018-07-12 00:18:34.969608 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1354] Level-0 flush table #11159: started
>2018-07-12 00:18:35.011753 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354715011741, "cf_name": "default", "job": 1354, "event": "table_file_creation", "file_number": 11159, "file_size": 85493, "table_properties": {"data_size": 83292, "index_size": 378, "filter_size": 904, "raw_key_size": 5406, "raw_average_key_size": 20, "raw_value_size": 79447, "raw_average_value_size": 296, "num_data_blocks": 12, "num_entries": 268, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:18:35.011764 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1354] Level-0 flush table #11159: 85493 bytes OK
>2018-07-12 00:18:35.070114 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.028284) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11159 started
>2018-07-12 00:18:35.070122 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.070081) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11159: memtable #1 done
>2018-07-12 00:18:35.070125 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.070095) EVENT_LOG_v1 {"time_micros": 1531354715070089, "job": 1354, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:18:35.070140 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.070106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:18:35.070145 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1354] Try to delete WAL files size 86166, prev total WAL file size 86166, number of live WAL files 2.
>
>2018-07-12 00:18:35.070258 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:18:35.070308 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.070302) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:18:35.070318 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1355] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:18:35.070323 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1354 Base level 0, inputs: [11159(83KB)], [11157(15MB)]
>
>2018-07-12 00:18:35.070336 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354715070327, "job": 1355, "event": "compaction_started", "files_L0": [11159], "files_L1": [11157], "score": -1, "input_data_size": 16591235}
>2018-07-12 00:18:35.245830 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1355] Generated table #11160: 2962 keys, 14631773 bytes
>2018-07-12 00:18:35.245880 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354715245852, "cf_name": "default", "job": 1355, "event": "table_file_creation", "file_number": 11160, "file_size": 14631773, "table_properties": {"data_size": 14517624, "index_size": 65843, "filter_size": 47379, "raw_key_size": 60966, "raw_average_key_size": 20, "raw_value_size": 14428929, "raw_average_value_size": 4871, "num_data_blocks": 1914, "num_entries": 2962, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:18:35.279010 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1355] Compacted 1@0 + 1@1 files to L1 => 14631773 bytes
>2018-07-12 00:18:35.312505 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.312438) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.5 rd, 70.1 wr, level 1, files in(1, 1) out(1) MB in(0.1, 15.7) out(14.0), read-write-amplify(365.2) write-amplify(171.1) OK, records in: 3470, records dropped: 508
>
>2018-07-12 00:18:35.312514 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:18:35.312460) EVENT_LOG_v1 {"time_micros": 1531354715312451, "job": 1355, "event": "compaction_finished", "compaction_time_micros": 208655, "output_level": 1, "num_output_files": 1, "total_output_size": 14631773, "num_input_records": 3470, "num_output_records": 2962, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:18:35.312618 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354715312614, "job": 1355, "event": "table_file_deletion", "file_number": 11159}
>2018-07-12 00:18:35.315847 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354715315844, "job": 1355, "event": "table_file_deletion", "file_number": 11157}
>2018-07-12 00:19:12.293254 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:20:12.293443 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:20:39.991218 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11161. Immutable memtables: 0.
>
>2018-07-12 00:20:39.991263 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1356] Syncing log #11158
>2018-07-12 00:20:40.007697 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:20:39.991253) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:20:40.007709 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1356] Flushing memtable with next log file: 11161
>
>2018-07-12 00:20:40.007735 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840007727, "job": 1356, "event": "flush_started", "num_memtables": 1, "num_entries": 696, "num_deletes": 250, "memory_usage": 946144}
>2018-07-12 00:20:40.007738 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1356] Level-0 flush table #11162: started
>2018-07-12 00:20:40.049890 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840049876, "cf_name": "default", "job": 1356, "event": "table_file_creation", "file_number": 11162, "file_size": 519683, "table_properties": {"data_size": 514230, "index_size": 2260, "filter_size": 2274, "raw_key_size": 7140, "raw_average_key_size": 21, "raw_value_size": 508114, "raw_average_value_size": 1544, "num_data_blocks": 71, "num_entries": 329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:20:40.049900 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1356] Level-0 flush table #11162: 519683 bytes OK
>2018-07-12 00:20:40.099912 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.066285) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11162 started
>2018-07-12 00:20:40.099921 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.099877) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11162: memtable #1 done
>2018-07-12 00:20:40.099923 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.099891) EVENT_LOG_v1 {"time_micros": 1531354840099885, "job": 1356, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:20:40.099925 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.099903) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:20:40.099944 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1356] Try to delete WAL files size 930961, prev total WAL file size 930961, number of live WAL files 2.
>
>2018-07-12 00:20:40.100238 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:20:40.100425 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 00:20:40.100432 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 116867.0 total, 617.2 interval
>Cumulative writes: 144K writes, 672K keys, 144K commit groups, 1.0 writes per commit group, ingest: 1.09 GB, 0.01 MB/s
>Cumulative WAL: 144K writes, 144K syncs, 1.00 writes per sync, written: 1.09 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 630 writes, 3450 keys, 630 commit groups, 1.0 writes per commit group, ingest: 4.44 MB, 0.01 MB/s
>Interval WAL: 630 writes, 630 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   507.50 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       679    0.077       0      0
>  L1      1/1   13.95 MB   0.0     10.7     0.9      9.8       9.8     -0.0       0.0  10.8     72.6     66.4       151       676    0.224   2433K   296K
> Sum      2/2   14.45 MB   0.0     10.7     0.9      9.8      10.7      0.9       0.0  11.8     53.9     53.9       204      1355    0.150   2433K   296K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  22.5     57.2     54.7         1        10    0.148     18K   2483
>Uptime(secs): 116867.0 total, 116867.0 interval
>Flush(GB): cumulative 0.908, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.72 GB write, 0.09 MB/s write, 10.72 GB read, 0.09 MB/s read, 203.7 seconds
>Interval compaction: 0.08 GB write, 0.00 MB/s write, 0.08 GB read, 0.00 MB/s read, 1.5 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   507.50 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       679    0.077       0      0
>  L1      1/1   13.95 MB   0.0     10.7     0.9      9.8       9.8     -0.0       0.0  10.8     72.6     66.4       151       676    0.224   2433K   296K
> Sum      2/2   14.45 MB   0.0     10.7     0.9      9.8      10.7      0.9       0.0  11.8     53.9     53.9       204      1355    0.150   2433K   296K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 116867.0 total, 116867.0 interval
>Flush(GB): cumulative 0.908, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.72 GB write, 0.09 MB/s write, 10.72 GB read, 0.09 MB/s read, 203.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 00:20:40.100454 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.100448) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:20:40.100461 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1357] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:20:40.100466 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1356 Base level 0, inputs: [11162(507KB)], [11160(13MB)]
>
>2018-07-12 00:20:40.100479 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840100470, "job": 1357, "event": "compaction_started", "files_L0": [11162], "files_L1": [11160], "score": -1, "input_data_size": 15151456}
>2018-07-12 00:20:40.267255 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1357] Generated table #11163: 2839 keys, 13830853 bytes
>2018-07-12 00:20:40.267305 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840267289, "cf_name": "default", "job": 1357, "event": "table_file_creation", "file_number": 11163, "file_size": 13830853, "table_properties": {"data_size": 13723868, "index_size": 61354, "filter_size": 44704, "raw_key_size": 58126, "raw_average_key_size": 20, "raw_value_size": 13640148, "raw_average_value_size": 4804, "num_data_blocks": 1789, "num_entries": 2839, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:20:40.300438 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1357] Compacted 1@0 + 1@1 files to L1 => 13830853 bytes
>2018-07-12 00:20:40.334034 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.333958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.8 rd, 69.2 wr, level 1, files in(1, 1) out(1) MB in(0.5, 14.0) out(13.2), read-write-amplify(55.8) write-amplify(26.6) OK, records in: 3291, records dropped: 452
>
>2018-07-12 00:20:40.334044 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:20:40.333983) EVENT_LOG_v1 {"time_micros": 1531354840333972, "job": 1357, "event": "compaction_finished", "compaction_time_micros": 199942, "output_level": 1, "num_output_files": 1, "total_output_size": 13830853, "num_input_records": 3291, "num_output_records": 2839, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:20:40.334240 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840334236, "job": 1357, "event": "table_file_deletion", "file_number": 11162}
>2018-07-12 00:20:40.336921 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531354840336918, "job": 1357, "event": "table_file_deletion", "file_number": 11160}
>2018-07-12 00:20:40.337036 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:21:12.293623 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:22:12.293797 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:23:12.293983 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:24:12.294170 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:25:12.294356 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:26:12.294537 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:26:51.157123 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11164. Immutable memtables: 0.
>
>2018-07-12 00:26:51.157167 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1358] Syncing log #11161
>2018-07-12 00:26:51.173695 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.157158) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:26:51.173706 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1358] Flushing memtable with next log file: 11164
>
>2018-07-12 00:26:51.173733 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211173724, "job": 1358, "event": "flush_started", "num_memtables": 1, "num_entries": 1558, "num_deletes": 251, "memory_usage": 2790968}
>2018-07-12 00:26:51.173736 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1358] Level-0 flush table #11165: started
>2018-07-12 00:26:51.240823 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211240808, "cf_name": "default", "job": 1358, "event": "table_file_creation", "file_number": 11165, "file_size": 2661626, "table_properties": {"data_size": 2639191, "index_size": 12357, "filter_size": 9154, "raw_key_size": 13008, "raw_average_key_size": 20, "raw_value_size": 2621592, "raw_average_value_size": 4161, "num_data_blocks": 374, "num_entries": 630, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:26:51.240847 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1358] Level-0 flush table #11165: 2661626 bytes OK
>2018-07-12 00:26:51.290824 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.257334) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11165 started
>2018-07-12 00:26:51.290833 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.290784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11165: memtable #1 done
>2018-07-12 00:26:51.290836 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.290799) EVENT_LOG_v1 {"time_micros": 1531355211290793, "job": 1358, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:26:51.290838 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.290813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:26:51.290845 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1358] Try to delete WAL files size 2759355, prev total WAL file size 2759355, number of live WAL files 2.
>
>2018-07-12 00:26:51.291566 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:26:51.291622 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.291614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:26:51.291634 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1359] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:26:51.291641 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1358 Base level 0, inputs: [11165(2599KB)], [11163(13MB)]
>
>2018-07-12 00:26:51.291657 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211291646, "job": 1359, "event": "compaction_started", "files_L0": [11165], "files_L1": [11163], "score": -1, "input_data_size": 16492479}
>2018-07-12 00:26:51.474815 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1359] Generated table #11166: 2961 keys, 14612548 bytes
>2018-07-12 00:26:51.474865 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211474837, "cf_name": "default", "job": 1359, "event": "table_file_creation", "file_number": 11166, "file_size": 14612548, "table_properties": {"data_size": 14498564, "index_size": 65739, "filter_size": 47318, "raw_key_size": 60938, "raw_average_key_size": 20, "raw_value_size": 14409958, "raw_average_value_size": 4866, "num_data_blocks": 1911, "num_entries": 2961, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:26:51.516439 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1359] Compacted 1@0 + 1@1 files to L1 => 14612548 bytes
>2018-07-12 00:26:51.549988 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.549920) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.4 rd, 65.0 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.9), read-write-amplify(11.7) write-amplify(5.5) OK, records in: 3469, records dropped: 508
>
>2018-07-12 00:26:51.549997 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:26:51.549941) EVENT_LOG_v1 {"time_micros": 1531355211549932, "job": 1359, "event": "compaction_finished", "compaction_time_micros": 224766, "output_level": 1, "num_output_files": 1, "total_output_size": 14612548, "num_input_records": 3469, "num_output_records": 2961, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:26:51.550633 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211550629, "job": 1359, "event": "table_file_deletion", "file_number": 11165}
>2018-07-12 00:26:51.553349 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355211553346, "job": 1359, "event": "table_file_deletion", "file_number": 11163}
>2018-07-12 00:27:12.294712 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:28:12.294898 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:29:00.187037 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11167. Immutable memtables: 0.
>
>2018-07-12 00:29:00.187080 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1360] Syncing log #11164
>2018-07-12 00:29:00.203581 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.187071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:29:00.203592 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1360] Flushing memtable with next log file: 11167
>
>2018-07-12 00:29:00.203619 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340203611, "job": 1360, "event": "flush_started", "num_memtables": 1, "num_entries": 703, "num_deletes": 250, "memory_usage": 973864}
>2018-07-12 00:29:00.203622 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1360] Level-0 flush table #11168: started
>2018-07-12 00:29:00.245789 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340245775, "cf_name": "default", "job": 1360, "event": "table_file_creation", "file_number": 11168, "file_size": 532737, "table_properties": {"data_size": 527178, "index_size": 2326, "filter_size": 2314, "raw_key_size": 7090, "raw_average_key_size": 21, "raw_value_size": 521068, "raw_average_value_size": 1593, "num_data_blocks": 73, "num_entries": 327, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:29:00.245799 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1360] Level-0 flush table #11168: 532737 bytes OK
>2018-07-12 00:29:00.295810 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.262332) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11168 started
>2018-07-12 00:29:00.295818 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.295774) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11168: memtable #1 done
>2018-07-12 00:29:00.295821 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.295789) EVENT_LOG_v1 {"time_micros": 1531355340295783, "job": 1360, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:29:00.295823 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.295800) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:29:00.295842 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1360] Try to delete WAL files size 958547, prev total WAL file size 958547, number of live WAL files 2.
>
>2018-07-12 00:29:00.296156 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:29:00.296219 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.296212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:29:00.296230 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1361] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:29:00.296236 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1360 Base level 0, inputs: [11168(520KB)], [11166(13MB)]
>
>2018-07-12 00:29:00.296249 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340296239, "job": 1361, "event": "compaction_started", "files_L0": [11168], "files_L1": [11166], "score": -1, "input_data_size": 15145285}
>2018-07-12 00:29:00.471510 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1361] Generated table #11169: 2840 keys, 13838311 bytes
>2018-07-12 00:29:00.471560 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340471532, "cf_name": "default", "job": 1361, "event": "table_file_creation", "file_number": 11169, "file_size": 13838311, "table_properties": {"data_size": 13731268, "index_size": 61387, "filter_size": 44729, "raw_key_size": 58146, "raw_average_key_size": 20, "raw_value_size": 13647511, "raw_average_value_size": 4805, "num_data_blocks": 1790, "num_entries": 2840, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:29:00.513013 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1361] Compacted 1@0 + 1@1 files to L1 => 13838311 bytes
>2018-07-12 00:29:00.546537 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.546470) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.9 rd, 63.8 wr, level 1, files in(1, 1) out(1) MB in(0.5, 13.9) out(13.2), read-write-amplify(54.4) write-amplify(26.0) OK, records in: 3288, records dropped: 448
>
>2018-07-12 00:29:00.546546 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:29:00.546492) EVENT_LOG_v1 {"time_micros": 1531355340546482, "job": 1361, "event": "compaction_finished", "compaction_time_micros": 216748, "output_level": 1, "num_output_files": 1, "total_output_size": 13838311, "num_input_records": 3288, "num_output_records": 2840, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:29:00.546736 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340546719, "job": 1361, "event": "table_file_deletion", "file_number": 11168}
>2018-07-12 00:29:00.549453 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355340549449, "job": 1361, "event": "table_file_deletion", "file_number": 11166}
>2018-07-12 00:29:00.549565 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:29:12.295071 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:30:12.295248 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:31:12.295424 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:32:12.295612 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:33:12.295787 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:34:12.295961 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:35:09.369720 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11170. Immutable memtables: 0.
>
>2018-07-12 00:35:09.369765 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1362] Syncing log #11167
>2018-07-12 00:35:09.386267 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.369755) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:35:09.386279 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1362] Flushing memtable with next log file: 11170
>
>2018-07-12 00:35:09.386294 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709386286, "job": 1362, "event": "flush_started", "num_memtables": 1, "num_entries": 1551, "num_deletes": 251, "memory_usage": 2775944}
>2018-07-12 00:35:09.386297 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1362] Level-0 flush table #11171: started
>2018-07-12 00:35:09.445150 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709445136, "cf_name": "default", "job": 1362, "event": "table_file_creation", "file_number": 11171, "file_size": 2647394, "table_properties": {"data_size": 2625073, "index_size": 12289, "filter_size": 9108, "raw_key_size": 12966, "raw_average_key_size": 20, "raw_value_size": 2607547, "raw_average_value_size": 4152, "num_data_blocks": 372, "num_entries": 628, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:35:09.445161 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1362] Level-0 flush table #11171: 2647394 bytes OK
>2018-07-12 00:35:09.495185 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.461709) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11171 started
>2018-07-12 00:35:09.495193 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.495141) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11171: memtable #1 done
>2018-07-12 00:35:09.495196 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.495155) EVENT_LOG_v1 {"time_micros": 1531355709495149, "job": 1362, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:35:09.495198 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.495168) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:35:09.495218 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1362] Try to delete WAL files size 2744586, prev total WAL file size 2744586, number of live WAL files 2.
>
>2018-07-12 00:35:09.495921 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:35:09.496118 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 00:35:09.496125 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 117736.4 total, 869.4 interval
>Cumulative writes: 144K writes, 676K keys, 144K commit groups, 1.0 writes per commit group, ingest: 1.10 GB, 0.01 MB/s
>Cumulative WAL: 144K writes, 144K syncs, 1.00 writes per sync, written: 1.10 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 876 writes, 3812 keys, 876 commit groups, 1.0 writes per commit group, ingest: 6.16 MB, 0.01 MB/s
>Interval WAL: 876 writes, 876 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.52 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       682    0.077       0      0
>  L1      1/1   13.20 MB   0.0     10.8     0.9      9.8       9.8     -0.0       0.0  10.8     72.6     66.4       152       679    0.224   2444K   297K
> Sum      2/2   15.72 MB   0.0     10.8     0.9      9.8      10.8      0.9       0.0  11.8     53.8     53.9       205      1361    0.150   2444K   297K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   8.2     51.9     53.4         1         6    0.143     10K   1408
>Uptime(secs): 117736.4 total, 117736.4 interval
>Flush(GB): cumulative 0.913, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.76 GB write, 0.09 MB/s write, 10.76 GB read, 0.09 MB/s read, 204.6 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.52 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       682    0.077       0      0
>  L1      1/1   13.20 MB   0.0     10.8     0.9      9.8       9.8     -0.0       0.0  10.8     72.6     66.4       152       679    0.224   2444K   297K
> Sum      2/2   15.72 MB   0.0     10.8     0.9      9.8      10.8      0.9       0.0  11.8     53.8     53.9       205      1361    0.150   2444K   297K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 117736.4 total, 117736.4 interval
>Flush(GB): cumulative 0.913, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.76 GB write, 0.09 MB/s write, 10.76 GB read, 0.09 MB/s read, 204.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 00:35:09.496149 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.496142) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:35:09.496154 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1363] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:35:09.496159 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1362 Base level 0, inputs: [11171(2585KB)], [11169(13MB)]
>
>2018-07-12 00:35:09.496172 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709496163, "job": 1363, "event": "compaction_started", "files_L0": [11171], "files_L1": [11169], "score": -1, "input_data_size": 16485705}
>2018-07-12 00:35:09.670896 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1363] Generated table #11172: 2960 keys, 14591361 bytes
>2018-07-12 00:35:09.670943 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709670916, "cf_name": "default", "job": 1363, "event": "table_file_creation", "file_number": 11172, "file_size": 14591361, "table_properties": {"data_size": 14477467, "index_size": 65702, "filter_size": 47265, "raw_key_size": 60916, "raw_average_key_size": 20, "raw_value_size": 14388901, "raw_average_value_size": 4861, "num_data_blocks": 1910, "num_entries": 2960, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:35:09.704056 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1363] Compacted 1@0 + 1@1 files to L1 => 14591361 bytes
>2018-07-12 00:35:09.737593 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.737525) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.3 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.9), read-write-amplify(11.7) write-amplify(5.5) OK, records in: 3468, records dropped: 508
>
>2018-07-12 00:35:09.737602 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:35:09.737546) EVENT_LOG_v1 {"time_micros": 1531355709737537, "job": 1363, "event": "compaction_finished", "compaction_time_micros": 207868, "output_level": 1, "num_output_files": 1, "total_output_size": 14591361, "num_input_records": 3468, "num_output_records": 2960, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:35:09.738165 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709738161, "job": 1363, "event": "table_file_deletion", "file_number": 11171}
>2018-07-12 00:35:09.740934 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355709740931, "job": 1363, "event": "table_file_deletion", "file_number": 11169}
>2018-07-12 00:35:12.296132 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:36:12.296322 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:37:12.296504 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:37:20.399394 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11173. Immutable memtables: 0.
>
>2018-07-12 00:37:20.399438 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1364] Syncing log #11170
>2018-07-12 00:37:20.415921 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.399428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:37:20.415933 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1364] Flushing memtable with next log file: 11173
>
>2018-07-12 00:37:20.415960 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840415951, "job": 1364, "event": "flush_started", "num_memtables": 1, "num_entries": 710, "num_deletes": 250, "memory_usage": 988872}
>2018-07-12 00:37:20.415963 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1364] Level-0 flush table #11174: started
>2018-07-12 00:37:20.458111 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840458097, "cf_name": "default", "job": 1364, "event": "table_file_creation", "file_number": 11174, "file_size": 540188, "table_properties": {"data_size": 534577, "index_size": 2357, "filter_size": 2335, "raw_key_size": 7110, "raw_average_key_size": 21, "raw_value_size": 528431, "raw_average_value_size": 1611, "num_data_blocks": 74, "num_entries": 328, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:37:20.458121 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1364] Level-0 flush table #11174: 540188 bytes OK
>2018-07-12 00:37:20.508109 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.474516) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11174 started
>2018-07-12 00:37:20.508117 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.508073) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11174: memtable #1 done
>2018-07-12 00:37:20.508119 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.508088) EVENT_LOG_v1 {"time_micros": 1531355840508081, "job": 1364, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:37:20.508133 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.508099) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:37:20.508140 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1364] Try to delete WAL files size 973316, prev total WAL file size 973316, number of live WAL files 2.
>
>2018-07-12 00:37:20.508428 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:37:20.508477 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.508470) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:37:20.508486 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1365] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:37:20.508492 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1364 Base level 0, inputs: [11174(527KB)], [11172(13MB)]
>
>2018-07-12 00:37:20.508506 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840508496, "job": 1365, "event": "compaction_started", "files_L0": [11174], "files_L1": [11172], "score": -1, "input_data_size": 15131549}
>2018-07-12 00:37:20.675487 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1365] Generated table #11175: 2841 keys, 13831363 bytes
>2018-07-12 00:37:20.675518 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840675504, "cf_name": "default", "job": 1365, "event": "table_file_creation", "file_number": 11175, "file_size": 13831363, "table_properties": {"data_size": 13724292, "index_size": 61418, "filter_size": 44726, "raw_key_size": 58166, "raw_average_key_size": 20, "raw_value_size": 13640499, "raw_average_value_size": 4801, "num_data_blocks": 1791, "num_entries": 2841, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:37:20.708685 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1365] Compacted 1@0 + 1@1 files to L1 => 13831363 bytes
>2018-07-12 00:37:20.742228 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.742152) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 75.6 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(0.5, 13.9) out(13.2), read-write-amplify(53.6) write-amplify(25.6) OK, records in: 3288, records dropped: 447
>
>2018-07-12 00:37:20.742237 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:37:20.742176) EVENT_LOG_v1 {"time_micros": 1531355840742164, "job": 1365, "event": "compaction_finished", "compaction_time_micros": 200163, "output_level": 1, "num_output_files": 1, "total_output_size": 13831363, "num_input_records": 3288, "num_output_records": 2841, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:37:20.742416 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840742411, "job": 1365, "event": "table_file_deletion", "file_number": 11174}
>2018-07-12 00:37:20.745143 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531355840745140, "job": 1365, "event": "table_file_deletion", "file_number": 11172}
>2018-07-12 00:37:20.745262 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:38:12.296676 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:39:12.296853 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:40:12.297026 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:41:12.297208 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:42:12.297388 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:43:12.297564 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8796 MB, avail 861 GB
>2018-07-12 00:43:27.615492 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11176. Immutable memtables: 0.
>
>2018-07-12 00:43:27.615534 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1366] Syncing log #11173
>2018-07-12 00:43:27.632034 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.615525) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:43:27.632045 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1366] Flushing memtable with next log file: 11176
>
>2018-07-12 00:43:27.632072 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207632064, "job": 1366, "event": "flush_started", "num_memtables": 1, "num_entries": 1544, "num_deletes": 251, "memory_usage": 2761168}
>2018-07-12 00:43:27.632076 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1366] Level-0 flush table #11177: started
>2018-07-12 00:43:27.690923 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207690908, "cf_name": "default", "job": 1366, "event": "table_file_creation", "file_number": 11177, "file_size": 2633156, "table_properties": {"data_size": 2610949, "index_size": 12221, "filter_size": 9062, "raw_key_size": 12924, "raw_average_key_size": 20, "raw_value_size": 2593502, "raw_average_value_size": 4142, "num_data_blocks": 370, "num_entries": 626, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:43:27.690934 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1366] Level-0 flush table #11177: 2633156 bytes OK
>2018-07-12 00:43:27.740926 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.707428) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11177 started
>2018-07-12 00:43:27.740934 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.740888) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11177: memtable #1 done
>2018-07-12 00:43:27.740937 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.740902) EVENT_LOG_v1 {"time_micros": 1531356207740896, "job": 1366, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:43:27.740952 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.740916) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:43:27.740958 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1366] Try to delete WAL files size 2729817, prev total WAL file size 2729817, number of live WAL files 2.
>
>2018-07-12 00:43:27.741711 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:43:27.741774 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.741767) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:43:27.741781 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1367] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:43:27.741787 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1366 Base level 0, inputs: [11177(2571KB)], [11175(13MB)]
>
>2018-07-12 00:43:27.741800 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207741790, "job": 1367, "event": "compaction_started", "files_L0": [11177], "files_L1": [11175], "score": -1, "input_data_size": 16464519}
>2018-07-12 00:43:27.916612 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1367] Generated table #11178: 2959 keys, 14591072 bytes
>2018-07-12 00:43:27.916653 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207916627, "cf_name": "default", "job": 1367, "event": "table_file_creation", "file_number": 11178, "file_size": 14591072, "table_properties": {"data_size": 14477178, "index_size": 65702, "filter_size": 47265, "raw_key_size": 60894, "raw_average_key_size": 20, "raw_value_size": 14388627, "raw_average_value_size": 4862, "num_data_blocks": 1910, "num_entries": 2959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:43:27.949805 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1367] Compacted 1@0 + 1@1 files to L1 => 14591072 bytes
>2018-07-12 00:43:27.983314 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.983246) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.2 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.9), read-write-amplify(11.8) write-amplify(5.5) OK, records in: 3467, records dropped: 508
>
>2018-07-12 00:43:27.983323 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:43:27.983267) EVENT_LOG_v1 {"time_micros": 1531356207983257, "job": 1367, "event": "compaction_finished", "compaction_time_micros": 207987, "output_level": 1, "num_output_files": 1, "total_output_size": 14591072, "num_input_records": 3467, "num_output_records": 2959, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:43:27.983885 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207983881, "job": 1367, "event": "table_file_deletion", "file_number": 11177}
>2018-07-12 00:43:27.986637 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356207986634, "job": 1367, "event": "table_file_deletion", "file_number": 11175}
>2018-07-12 00:44:12.297743 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:45:12.297929 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:45:40.604687 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11179. Immutable memtables: 0.
>
>2018-07-12 00:45:40.604765 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1368] Syncing log #11176
>2018-07-12 00:45:40.621388 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.604754) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:45:40.621399 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1368] Flushing memtable with next log file: 11179
>
>2018-07-12 00:45:40.621413 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340621405, "job": 1368, "event": "flush_started", "num_memtables": 1, "num_entries": 717, "num_deletes": 250, "memory_usage": 1003464}
>2018-07-12 00:45:40.621416 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1368] Level-0 flush table #11180: started
>2018-07-12 00:45:40.663588 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340663575, "cf_name": "default", "job": 1368, "event": "table_file_creation", "file_number": 11180, "file_size": 547648, "table_properties": {"data_size": 541979, "index_size": 2390, "filter_size": 2360, "raw_key_size": 7130, "raw_average_key_size": 21, "raw_value_size": 535794, "raw_average_value_size": 1628, "num_data_blocks": 75, "num_entries": 329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:45:40.663599 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1368] Level-0 flush table #11180: 547648 bytes OK
>2018-07-12 00:45:40.713551 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.680158) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11180 started
>2018-07-12 00:45:40.713559 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.713502) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11180: memtable #1 done
>2018-07-12 00:45:40.713562 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.713517) EVENT_LOG_v1 {"time_micros": 1531356340713511, "job": 1368, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:45:40.713564 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.713541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:45:40.713570 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1368] Try to delete WAL files size 988085, prev total WAL file size 988085, number of live WAL files 2.
>
>2018-07-12 00:45:40.713842 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:45:40.714024 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 00:45:40.714031 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 118367.6 total, 631.2 interval
>Cumulative writes: 145K writes, 679K keys, 145K commit groups, 1.0 writes per commit group, ingest: 1.10 GB, 0.01 MB/s
>Cumulative WAL: 145K writes, 145K syncs, 1.00 writes per sync, written: 1.10 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 636 writes, 2971 keys, 636 commit groups, 1.0 writes per commit group, ingest: 4.47 MB, 0.01 MB/s
>Interval WAL: 636 writes, 636 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   534.81 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       685    0.077       0      0
>  L1      1/1   13.92 MB   0.0     10.8     0.9      9.9       9.9     -0.0       0.0  10.8     72.6     66.4       152       682    0.224   2454K   299K
> Sum      2/2   14.44 MB   0.0     10.8     0.9      9.9      10.8      0.9       0.0  11.8     53.9     53.9       205      1367    0.150   2454K   299K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.6     56.7     55.1         1         6    0.135     10K   1463
>Uptime(secs): 118367.6 total, 118367.6 interval
>Flush(GB): cumulative 0.916, interval 0.003
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.80 GB write, 0.09 MB/s write, 10.80 GB read, 0.09 MB/s read, 205.4 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   534.81 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.7        53       685    0.077       0      0
>  L1      1/1   13.92 MB   0.0     10.8     0.9      9.9       9.9     -0.0       0.0  10.8     72.6     66.4       152       682    0.224   2454K   299K
> Sum      2/2   14.44 MB   0.0     10.8     0.9      9.9      10.8      0.9       0.0  11.8     53.9     53.9       205      1367    0.150   2454K   299K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 118367.6 total, 118367.6 interval
>Flush(GB): cumulative 0.916, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.80 GB write, 0.09 MB/s write, 10.80 GB read, 0.09 MB/s read, 205.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 00:45:40.714051 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.714044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:45:40.714055 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1369] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:45:40.714061 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1368 Base level 0, inputs: [11180(534KB)], [11178(13MB)]
>
>2018-07-12 00:45:40.714073 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340714064, "job": 1369, "event": "compaction_started", "files_L0": [11180], "files_L1": [11178], "score": -1, "input_data_size": 15138720}
>2018-07-12 00:45:40.889338 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1369] Generated table #11181: 2842 keys, 13845306 bytes
>2018-07-12 00:45:40.889370 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340889355, "cf_name": "default", "job": 1369, "event": "table_file_creation", "file_number": 11181, "file_size": 13845306, "table_properties": {"data_size": 13738124, "index_size": 61487, "filter_size": 44768, "raw_key_size": 58186, "raw_average_key_size": 20, "raw_value_size": 13654270, "raw_average_value_size": 4804, "num_data_blocks": 1793, "num_entries": 2842, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:45:40.930830 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1369] Compacted 1@0 + 1@1 files to L1 => 13845306 bytes
>2018-07-12 00:45:40.972573 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.972499) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 69.8 rd, 63.9 wr, level 1, files in(1, 1) out(1) MB in(0.5, 13.9) out(13.2), read-write-amplify(52.9) write-amplify(25.3) OK, records in: 3288, records dropped: 446
>
>2018-07-12 00:45:40.972582 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:45:40.972520) EVENT_LOG_v1 {"time_micros": 1531356340972511, "job": 1369, "event": "compaction_finished", "compaction_time_micros": 216740, "output_level": 1, "num_output_files": 1, "total_output_size": 13845306, "num_input_records": 3288, "num_output_records": 2842, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:45:40.972739 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340972734, "job": 1369, "event": "table_file_deletion", "file_number": 11180}
>2018-07-12 00:45:40.975452 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356340975449, "job": 1369, "event": "table_file_deletion", "file_number": 11178}
>2018-07-12 00:45:40.975505 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:46:12.298114 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:47:12.298293 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:48:12.298476 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:49:12.298653 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:49:19.166882 7f29149e4700  1 mon.magna102@0(leader).log v22271 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22271)
>2018-07-12 00:50:12.298824 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:51:12.299016 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:51:43.796891 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11182. Immutable memtables: 0.
>
>2018-07-12 00:51:43.796935 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1370] Syncing log #11179
>2018-07-12 00:51:43.813473 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.796925) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:51:43.813485 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1370] Flushing memtable with next log file: 11182
>
>2018-07-12 00:51:43.813498 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356703813491, "job": 1370, "event": "flush_started", "num_memtables": 1, "num_entries": 1538, "num_deletes": 251, "memory_usage": 2749816}
>2018-07-12 00:51:43.813501 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1370] Level-0 flush table #11183: started
>2018-07-12 00:51:43.872372 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356703872358, "cf_name": "default", "job": 1370, "event": "table_file_creation", "file_number": 11183, "file_size": 2623330, "table_properties": {"data_size": 2601227, "index_size": 12151, "filter_size": 9028, "raw_key_size": 12952, "raw_average_key_size": 20, "raw_value_size": 2583784, "raw_average_value_size": 4120, "num_data_blocks": 368, "num_entries": 627, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:51:43.872383 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1370] Level-0 flush table #11183: 2623330 bytes OK
>2018-07-12 00:51:43.922401 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.888907) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11183 started
>2018-07-12 00:51:43.922409 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.922363) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11183: memtable #1 done
>2018-07-12 00:51:43.922412 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.922378) EVENT_LOG_v1 {"time_micros": 1531356703922371, "job": 1370, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:51:43.922414 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.922391) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:51:43.922420 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1370] Try to delete WAL files size 2718849, prev total WAL file size 2718849, number of live WAL files 2.
>
>2018-07-12 00:51:43.923099 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:51:43.923154 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:51:43.923147) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:51:43.923163 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1371] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:51:43.923169 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1370 Base level 0, inputs: [11183(2561KB)], [11181(13MB)]
>
>2018-07-12 00:51:43.923207 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356703923172, "job": 1371, "event": "compaction_started", "files_L0": [11183], "files_L1": [11181], "score": -1, "input_data_size": 16468636}
>2018-07-12 00:51:44.098133 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1371] Generated table #11184: 2959 keys, 14588631 bytes
>2018-07-12 00:51:44.098193 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356704098153, "cf_name": "default", "job": 1371, "event": "table_file_creation", "file_number": 11184, "file_size": 14588631, "table_properties": {"data_size": 14474782, "index_size": 65667, "filter_size": 47255, "raw_key_size": 60891, "raw_average_key_size": 20, "raw_value_size": 14386256, "raw_average_value_size": 4861, "num_data_blocks": 1909, "num_entries": 2959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:51:44.131266 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1371] Compacted 1@0 + 1@1 files to L1 => 14588631 bytes
>2018-07-12 00:51:44.164826 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:51:44.164759) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.2 rd, 70.1 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.9), read-write-amplify(11.8) write-amplify(5.6) OK, records in: 3469, records dropped: 510
>
>2018-07-12 00:51:44.164835 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:51:44.164781) EVENT_LOG_v1 {"time_micros": 1531356704164771, "job": 1371, "event": "compaction_finished", "compaction_time_micros": 208044, "output_level": 1, "num_output_files": 1, "total_output_size": 14588631, "num_input_records": 3469, "num_output_records": 2959, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:51:44.165417 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356704165412, "job": 1371, "event": "table_file_deletion", "file_number": 11183}
>2018-07-12 00:51:44.168131 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356704168128, "job": 1371, "event": "table_file_deletion", "file_number": 11181}
>2018-07-12 00:52:12.299205 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:53:06.830570 7f29149e4700  1 mon.magna102@0(leader).log v22272 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22272)
>2018-07-12 00:53:12.299382 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:54:00.835931 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11185. Immutable memtables: 0.
>
>2018-07-12 00:54:00.835975 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1372] Syncing log #11182
>2018-07-12 00:54:00.852503 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.835966) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:54:00.852515 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1372] Flushing memtable with next log file: 11185
>
>2018-07-12 00:54:00.852528 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356840852521, "job": 1372, "event": "flush_started", "num_memtables": 1, "num_entries": 739, "num_deletes": 250, "memory_usage": 1053856}
>2018-07-12 00:54:00.852531 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1372] Level-0 flush table #11186: started
>2018-07-12 00:54:00.894694 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356840894680, "cf_name": "default", "job": 1372, "event": "table_file_creation", "file_number": 11186, "file_size": 582047, "table_properties": {"data_size": 576186, "index_size": 2482, "filter_size": 2460, "raw_key_size": 7282, "raw_average_key_size": 21, "raw_value_size": 569814, "raw_average_value_size": 1695, "num_data_blocks": 78, "num_entries": 336, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 00:54:00.894704 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1372] Level-0 flush table #11186: 582047 bytes OK
>2018-07-12 00:54:00.944737 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.911258) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11186 started
>2018-07-12 00:54:00.944745 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.944701) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11186: memtable #1 done
>2018-07-12 00:54:00.944747 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.944716) EVENT_LOG_v1 {"time_micros": 1531356840944709, "job": 1372, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:54:00.944749 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.944727) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:54:00.944756 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1372] Try to delete WAL files size 1037998, prev total WAL file size 1037998, number of live WAL files 2.
>
>2018-07-12 00:54:00.945038 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:54:00.945090 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:54:00.945083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 00:54:00.945099 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1373] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:54:00.945105 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1372 Base level 0, inputs: [11186(568KB)], [11184(13MB)]
>
>2018-07-12 00:54:00.945118 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356840945108, "job": 1373, "event": "compaction_started", "files_L0": [11186], "files_L1": [11184], "score": -1, "input_data_size": 15170678}
>2018-07-12 00:54:01.120444 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1373] Generated table #11187: 2849 keys, 13890829 bytes
>2018-07-12 00:54:01.120492 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356841120477, "cf_name": "default", "job": 1373, "event": "table_file_creation", "file_number": 11187, "file_size": 13890829, "table_properties": {"data_size": 13783337, "index_size": 61655, "filter_size": 44910, "raw_key_size": 58328, "raw_average_key_size": 20, "raw_value_size": 13699267, "raw_average_value_size": 4808, "num_data_blocks": 1798, "num_entries": 2849, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:54:01.153595 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1373] Compacted 1@0 + 1@1 files to L1 => 13890829 bytes
>2018-07-12 00:54:01.187105 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:54:01.187035) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 72.8 rd, 66.6 wr, level 1, files in(1, 1) out(1) MB in(0.6, 13.9) out(13.2), read-write-amplify(49.9) write-amplify(23.9) OK, records in: 3295, records dropped: 446
>
>2018-07-12 00:54:01.187113 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:54:01.187057) EVENT_LOG_v1 {"time_micros": 1531356841187047, "job": 1373, "event": "compaction_finished", "compaction_time_micros": 208460, "output_level": 1, "num_output_files": 1, "total_output_size": 13890829, "num_input_records": 3295, "num_output_records": 2849, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:54:01.187290 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356841187286, "job": 1373, "event": "table_file_deletion", "file_number": 11186}
>2018-07-12 00:54:01.189939 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531356841189936, "job": 1373, "event": "table_file_deletion", "file_number": 11184}
>2018-07-12 00:54:01.189990 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:54:12.299569 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:55:12.299742 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:56:00.915133 7f29149e4700  1 mon.magna102@0(leader).log v22273 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22273)
>2018-07-12 00:56:12.299926 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:57:12.300099 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:58:12.300272 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:59:12.300450 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 00:59:58.010642 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11188. Immutable memtables: 0.
>
>2018-07-12 00:59:58.010686 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1374] Syncing log #11185
>2018-07-12 00:59:58.027206 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.010677) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 00:59:58.027217 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1374] Flushing memtable with next log file: 11188
>
>2018-07-12 00:59:58.027231 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198027223, "job": 1374, "event": "flush_started", "num_memtables": 1, "num_entries": 1517, "num_deletes": 251, "memory_usage": 2721024}
>2018-07-12 00:59:58.027234 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1374] Level-0 flush table #11189: started
>2018-07-12 00:59:58.086120 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198086106, "cf_name": "default", "job": 1374, "event": "table_file_creation", "file_number": 11189, "file_size": 2596963, "table_properties": {"data_size": 2575168, "index_size": 11949, "filter_size": 8922, "raw_key_size": 12826, "raw_average_key_size": 20, "raw_value_size": 2557955, "raw_average_value_size": 4119, "num_data_blocks": 362, "num_entries": 621, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 00:59:58.086131 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1374] Level-0 flush table #11189: 2596963 bytes OK
>2018-07-12 00:59:58.144316 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.102611) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11189 started
>2018-07-12 00:59:58.144324 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.144278) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11189: memtable #1 done
>2018-07-12 00:59:58.144327 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.144293) EVENT_LOG_v1 {"time_micros": 1531357198144286, "job": 1374, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 00:59:58.144330 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.144306) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 00:59:58.144336 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1374] Try to delete WAL files size 2690848, prev total WAL file size 2690848, number of live WAL files 2.
>
>2018-07-12 00:59:58.144995 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 00:59:58.145191 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 00:59:58.145198 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 119225.0 total, 857.4 interval
>Cumulative writes: 146K writes, 683K keys, 146K commit groups, 1.0 writes per commit group, ingest: 1.11 GB, 0.01 MB/s
>Cumulative WAL: 146K writes, 146K syncs, 1.00 writes per sync, written: 1.11 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 870 writes, 3794 keys, 870 commit groups, 1.0 writes per commit group, ingest: 6.15 MB, 0.01 MB/s
>Interval WAL: 870 writes, 870 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.48 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        53       688    0.077       0      0
>  L1      1/1   13.25 MB   0.0     10.8     0.9      9.9       9.9     -0.0       0.0  10.8     72.6     66.4       153       685    0.223   2464K   300K
> Sum      2/2   15.72 MB   0.0     10.8     0.9      9.9      10.8      0.9       0.0  11.8     53.9     53.9       206      1373    0.150   2464K   300K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   8.3     52.9     54.5         1         6    0.140     10K   1402
>Uptime(secs): 119225.0 total, 119225.0 interval
>Flush(GB): cumulative 0.922, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.85 GB write, 0.09 MB/s write, 10.85 GB read, 0.09 MB/s read, 206.3 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.48 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        53       688    0.077       0      0
>  L1      1/1   13.25 MB   0.0     10.8     0.9      9.9       9.9     -0.0       0.0  10.8     72.6     66.4       153       685    0.223   2464K   300K
> Sum      2/2   15.72 MB   0.0     10.8     0.9      9.9      10.8      0.9       0.0  11.8     53.9     53.9       206      1373    0.150   2464K   300K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 119225.0 total, 119225.0 interval
>Flush(GB): cumulative 0.922, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.85 GB write, 0.09 MB/s write, 10.85 GB read, 0.09 MB/s read, 206.3 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 00:59:58.145218 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.145212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 00:59:58.145223 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1375] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 00:59:58.145228 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1374 Base level 0, inputs: [11189(2536KB)], [11187(13MB)]
>
>2018-07-12 00:59:58.145241 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198145231, "job": 1375, "event": "compaction_started", "files_L0": [11189], "files_L1": [11187], "score": -1, "input_data_size": 16487792}
>2018-07-12 00:59:58.311808 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1375] Generated table #11190: 2960 keys, 14607785 bytes
>2018-07-12 00:59:58.311857 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198311829, "cf_name": "default", "job": 1375, "event": "table_file_creation", "file_number": 11190, "file_size": 14607785, "table_properties": {"data_size": 14493938, "index_size": 65633, "filter_size": 47287, "raw_key_size": 60907, "raw_average_key_size": 20, "raw_value_size": 14405424, "raw_average_value_size": 4866, "num_data_blocks": 1908, "num_entries": 2960, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 00:59:58.345022 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1375] Compacted 1@0 + 1@1 files to L1 => 14607785 bytes
>2018-07-12 00:59:58.386874 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.386806) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 82.5 rd, 73.1 wr, level 1, files in(1, 1) out(1) MB in(2.5, 13.2) out(13.9), read-write-amplify(12.0) write-amplify(5.6) OK, records in: 3470, records dropped: 510
>
>2018-07-12 00:59:58.386883 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-00:59:58.386827) EVENT_LOG_v1 {"time_micros": 1531357198386818, "job": 1375, "event": "compaction_finished", "compaction_time_micros": 199767, "output_level": 1, "num_output_files": 1, "total_output_size": 14607785, "num_input_records": 3470, "num_output_records": 2960, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 00:59:58.387408 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198387404, "job": 1375, "event": "table_file_deletion", "file_number": 11189}
>2018-07-12 00:59:58.390112 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357198390109, "job": 1375, "event": "table_file_deletion", "file_number": 11187}
>2018-07-12 01:00:12.300635 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:00:49.150995 7f29149e4700  1 mon.magna102@0(leader).log v22274 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22274)
>2018-07-12 01:01:12.300807 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:01:57.066017 7f29149e4700  1 mon.magna102@0(leader).log v22275 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22275)
>2018-07-12 01:02:12.300988 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:02:21.051484 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11191. Immutable memtables: 0.
>
>2018-07-12 01:02:21.051561 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1376] Syncing log #11188
>2018-07-12 01:02:21.067962 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.051551) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:02:21.067973 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1376] Flushing memtable with next log file: 11191
>
>2018-07-12 01:02:21.067987 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341067979, "job": 1376, "event": "flush_started", "num_memtables": 1, "num_entries": 768, "num_deletes": 250, "memory_usage": 1118488}
>2018-07-12 01:02:21.067989 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1376] Level-0 flush table #11192: started
>2018-07-12 01:02:21.110154 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341110140, "cf_name": "default", "job": 1376, "event": "table_file_creation", "file_number": 11192, "file_size": 622510, "table_properties": {"data_size": 616362, "index_size": 2646, "filter_size": 2583, "raw_key_size": 7403, "raw_average_key_size": 21, "raw_value_size": 609787, "raw_average_value_size": 1783, "num_data_blocks": 83, "num_entries": 342, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:02:21.110165 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1376] Level-0 flush table #11192: 622510 bytes OK
>2018-07-12 01:02:21.160211 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.126711) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11192 started
>2018-07-12 01:02:21.160219 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.160168) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11192: memtable #1 done
>2018-07-12 01:02:21.160221 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.160189) EVENT_LOG_v1 {"time_micros": 1531357341160177, "job": 1376, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:02:21.160223 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.160201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:02:21.160230 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1376] Try to delete WAL files size 1102183, prev total WAL file size 1102183, number of live WAL files 2.
>
>2018-07-12 01:02:21.160535 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:02:21.160587 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.160580) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:02:21.160596 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1377] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:02:21.160601 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1376 Base level 0, inputs: [11192(607KB)], [11190(13MB)]
>
>2018-07-12 01:02:21.160615 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341160605, "job": 1377, "event": "compaction_started", "files_L0": [11192], "files_L1": [11190], "score": -1, "input_data_size": 15230295}
>2018-07-12 01:02:21.335788 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1377] Generated table #11193: 2859 keys, 13962081 bytes
>2018-07-12 01:02:21.335832 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341335805, "cf_name": "default", "job": 1377, "event": "table_file_creation", "file_number": 11193, "file_size": 13962081, "table_properties": {"data_size": 13854142, "index_size": 61895, "filter_size": 45117, "raw_key_size": 58531, "raw_average_key_size": 20, "raw_value_size": 13769766, "raw_average_value_size": 4816, "num_data_blocks": 1805, "num_entries": 2859, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:02:21.369108 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1377] Compacted 1@0 + 1@1 files to L1 => 13962081 bytes
>2018-07-12 01:02:21.402618 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.402549) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.1 rd, 67.0 wr, level 1, files in(1, 1) out(1) MB in(0.6, 13.9) out(13.3), read-write-amplify(46.9) write-amplify(22.4) OK, records in: 3302, records dropped: 443
>
>2018-07-12 01:02:21.402627 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:02:21.402571) EVENT_LOG_v1 {"time_micros": 1531357341402561, "job": 1377, "event": "compaction_finished", "compaction_time_micros": 208478, "output_level": 1, "num_output_files": 1, "total_output_size": 13962081, "num_input_records": 3302, "num_output_records": 2859, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:02:21.402801 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341402797, "job": 1377, "event": "table_file_deletion", "file_number": 11192}
>2018-07-12 01:02:21.405497 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357341405494, "job": 1377, "event": "table_file_deletion", "file_number": 11190}
>2018-07-12 01:02:21.405549 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:02:37.080640 7f29149e4700  1 mon.magna102@0(leader).log v22276 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22276)
>2018-07-12 01:03:12.301170 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:04:12.301366 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:05:12.301546 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:06:12.301719 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:07:12.301908 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:08:10.258854 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11194. Immutable memtables: 0.
>
>2018-07-12 01:08:10.258899 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1378] Syncing log #11191
>2018-07-12 01:08:10.275297 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.258889) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:08:10.275308 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1378] Flushing memtable with next log file: 11194
>
>2018-07-12 01:08:10.275336 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690275327, "job": 1378, "event": "flush_started", "num_memtables": 1, "num_entries": 1489, "num_deletes": 251, "memory_usage": 2644480}
>2018-07-12 01:08:10.275339 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1378] Level-0 flush table #11195: started
>2018-07-12 01:08:10.334214 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690334200, "cf_name": "default", "job": 1378, "event": "table_file_creation", "file_number": 11195, "file_size": 2522340, "table_properties": {"data_size": 2501033, "index_size": 11677, "filter_size": 8706, "raw_key_size": 12658, "raw_average_key_size": 20, "raw_value_size": 2484121, "raw_average_value_size": 4052, "num_data_blocks": 354, "num_entries": 613, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:08:10.334225 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1378] Level-0 flush table #11195: 2522340 bytes OK
>2018-07-12 01:08:10.384211 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.350707) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11195 started
>2018-07-12 01:08:10.384219 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.384166) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11195: memtable #1 done
>2018-07-12 01:08:10.384222 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.384187) EVENT_LOG_v1 {"time_micros": 1531357690384174, "job": 1378, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:08:10.384236 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.384201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:08:10.384243 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1378] Try to delete WAL files size 2614118, prev total WAL file size 2614118, number of live WAL files 2.
>
>2018-07-12 01:08:10.384908 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:08:10.384971 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.384964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:08:10.384978 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1379] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:08:10.384984 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1378 Base level 0, inputs: [11195(2463KB)], [11193(13MB)]
>
>2018-07-12 01:08:10.384998 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690384987, "job": 1379, "event": "compaction_started", "files_L0": [11195], "files_L1": [11193], "score": -1, "input_data_size": 16484421}
>2018-07-12 01:08:10.568284 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1379] Generated table #11196: 2962 keys, 14602501 bytes
>2018-07-12 01:08:10.568322 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690568307, "cf_name": "default", "job": 1379, "event": "table_file_creation", "file_number": 11196, "file_size": 14602501, "table_properties": {"data_size": 14488695, "index_size": 65602, "filter_size": 47277, "raw_key_size": 60942, "raw_average_key_size": 20, "raw_value_size": 14400179, "raw_average_value_size": 4861, "num_data_blocks": 1907, "num_entries": 2962, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:08:10.601424 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1379] Compacted 1@0 + 1@1 files to L1 => 14602501 bytes
>2018-07-12 01:08:10.634815 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.634746) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.2 rd, 67.5 wr, level 1, files in(1, 1) out(1) MB in(2.4, 13.3) out(13.9), read-write-amplify(12.3) write-amplify(5.8) OK, records in: 3472, records dropped: 510
>
>2018-07-12 01:08:10.634824 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:08:10.634768) EVENT_LOG_v1 {"time_micros": 1531357690634758, "job": 1379, "event": "compaction_finished", "compaction_time_micros": 216408, "output_level": 1, "num_output_files": 1, "total_output_size": 14602501, "num_input_records": 3472, "num_output_records": 2962, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:08:10.635378 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690635374, "job": 1379, "event": "table_file_deletion", "file_number": 11195}
>2018-07-12 01:08:10.638174 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357690638171, "job": 1379, "event": "table_file_deletion", "file_number": 11193}
>2018-07-12 01:08:12.302086 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:09:12.302262 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:10:12.302449 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:10:41.258747 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11197. Immutable memtables: 0.
>
>2018-07-12 01:10:41.258790 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1380] Syncing log #11194
>2018-07-12 01:10:41.275295 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.258781) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:10:41.275307 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1380] Flushing memtable with next log file: 11197
>
>2018-07-12 01:10:41.275320 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841275313, "job": 1380, "event": "flush_started", "num_memtables": 1, "num_entries": 780, "num_deletes": 250, "memory_usage": 1137888}
>2018-07-12 01:10:41.275323 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1380] Level-0 flush table #11198: started
>2018-07-12 01:10:41.317465 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841317438, "cf_name": "default", "job": 1380, "event": "table_file_creation", "file_number": 11198, "file_size": 614744, "table_properties": {"data_size": 608577, "index_size": 2675, "filter_size": 2573, "raw_key_size": 7310, "raw_average_key_size": 21, "raw_value_size": 602061, "raw_average_value_size": 1781, "num_data_blocks": 84, "num_entries": 338, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:10:41.317476 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1380] Level-0 flush table #11198: 614744 bytes OK
>2018-07-12 01:10:41.367565 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.334145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11198 started
>2018-07-12 01:10:41.367573 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.367529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11198: memtable #1 done
>2018-07-12 01:10:41.367576 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.367544) EVENT_LOG_v1 {"time_micros": 1531357841367538, "job": 1380, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:10:41.367578 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.367556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:10:41.367584 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1380] Try to delete WAL files size 1121006, prev total WAL file size 1121006, number of live WAL files 2.
>
>2018-07-12 01:10:41.367973 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:10:41.368154 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 01:10:41.368160 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 119868.2 total, 643.2 interval
>Cumulative writes: 147K writes, 686K keys, 147K commit groups, 1.0 writes per commit group, ingest: 1.11 GB, 0.01 MB/s
>Cumulative WAL: 147K writes, 147K syncs, 1.00 writes per sync, written: 1.11 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 654 writes, 3037 keys, 654 commit groups, 1.0 writes per commit group, ingest: 4.61 MB, 0.01 MB/s
>Interval WAL: 654 writes, 654 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   600.34 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        53       691    0.077       0      0
>  L1      1/1   13.93 MB   0.0     10.9     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       154       688    0.223   2474K   301K
> Sum      2/2   14.51 MB   0.0     10.9     0.9     10.0      10.9      0.9       0.0  11.8     53.9     53.9       207      1379    0.150   2474K   301K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.5     56.2     54.7         1         6    0.136     10K   1463
>Uptime(secs): 119868.2 total, 119868.2 interval
>Flush(GB): cumulative 0.925, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.89 GB write, 0.09 MB/s write, 10.89 GB read, 0.09 MB/s read, 207.1 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   600.34 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        53       691    0.077       0      0
>  L1      1/1   13.93 MB   0.0     10.9     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       154       688    0.223   2474K   301K
> Sum      2/2   14.51 MB   0.0     10.9     0.9     10.0      10.9      0.9       0.0  11.8     53.9     53.9       207      1379    0.150   2474K   301K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 119868.2 total, 119868.2 interval
>Flush(GB): cumulative 0.925, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.89 GB write, 0.09 MB/s write, 10.89 GB read, 0.09 MB/s read, 207.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 01:10:41.368192 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.368175) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:10:41.368210 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1381] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:10:41.368216 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1380 Base level 0, inputs: [11198(600KB)], [11196(13MB)]
>
>2018-07-12 01:10:41.368229 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841368219, "job": 1381, "event": "compaction_started", "files_L0": [11198], "files_L1": [11196], "score": -1, "input_data_size": 15217245}
>2018-07-12 01:10:41.543283 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1381] Generated table #11199: 2863 keys, 13984842 bytes
>2018-07-12 01:10:41.543321 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841543305, "cf_name": "default", "job": 1381, "event": "table_file_creation", "file_number": 11199, "file_size": 13984842, "table_properties": {"data_size": 13876730, "index_size": 61995, "filter_size": 45190, "raw_key_size": 58612, "raw_average_key_size": 20, "raw_value_size": 13792227, "raw_average_value_size": 4817, "num_data_blocks": 1808, "num_entries": 2863, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:10:41.584783 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1381] Compacted 1@0 + 1@1 files to L1 => 13984842 bytes
>2018-07-12 01:10:41.618290 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.618219) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 70.3 rd, 64.6 wr, level 1, files in(1, 1) out(1) MB in(0.6, 13.9) out(13.3), read-write-amplify(47.5) write-amplify(22.7) OK, records in: 3300, records dropped: 437
>
>2018-07-12 01:10:41.618298 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:10:41.618241) EVENT_LOG_v1 {"time_micros": 1531357841618231, "job": 1381, "event": "compaction_finished", "compaction_time_micros": 216540, "output_level": 1, "num_output_files": 1, "total_output_size": 13984842, "num_input_records": 3300, "num_output_records": 2863, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:10:41.618473 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841618469, "job": 1381, "event": "table_file_deletion", "file_number": 11198}
>2018-07-12 01:10:41.621206 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531357841621203, "job": 1381, "event": "table_file_deletion", "file_number": 11196}
>2018-07-12 01:10:41.621258 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:11:12.302636 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:12:12.302811 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:13:12.302985 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:14:12.303158 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:15:12.315899 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:16:12.337941 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:16:28.457494 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11200. Immutable memtables: 0.
>
>2018-07-12 01:16:28.457537 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1382] Syncing log #11197
>2018-07-12 01:16:28.474064 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.457528) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:16:28.474076 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1382] Flushing memtable with next log file: 11200
>
>2018-07-12 01:16:28.474090 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188474082, "job": 1382, "event": "flush_started", "num_memtables": 1, "num_entries": 1474, "num_deletes": 251, "memory_usage": 2612288}
>2018-07-12 01:16:28.474093 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1382] Level-0 flush table #11201: started
>2018-07-12 01:16:28.532959 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188532944, "cf_name": "default", "job": 1382, "event": "table_file_creation", "file_number": 11201, "file_size": 2490814, "table_properties": {"data_size": 2469739, "index_size": 11545, "filter_size": 8606, "raw_key_size": 12504, "raw_average_key_size": 20, "raw_value_size": 2453052, "raw_average_value_size": 4047, "num_data_blocks": 350, "num_entries": 606, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:16:28.532969 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1382] Level-0 flush table #11201: 2490814 bytes OK
>2018-07-12 01:16:28.582940 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.549467) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11201 started
>2018-07-12 01:16:28.582949 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.582903) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11201: memtable #1 done
>2018-07-12 01:16:28.582951 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.582917) EVENT_LOG_v1 {"time_micros": 1531358188582911, "job": 1382, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:16:28.582953 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.582930) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:16:28.582959 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1382] Try to delete WAL files size 2582127, prev total WAL file size 2582127, number of live WAL files 2.
>
>2018-07-12 01:16:28.583620 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:16:28.583675 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.583668) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:16:28.583684 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1383] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:16:28.583690 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1382 Base level 0, inputs: [11201(2432KB)], [11199(13MB)]
>
>2018-07-12 01:16:28.583703 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188583693, "job": 1383, "event": "compaction_started", "files_L0": [11201], "files_L1": [11199], "score": -1, "input_data_size": 16475656}
>2018-07-12 01:16:28.766939 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1383] Generated table #11202: 2961 keys, 14582821 bytes
>2018-07-12 01:16:28.766983 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188766956, "cf_name": "default", "job": 1383, "event": "table_file_creation", "file_number": 11202, "file_size": 14582821, "table_properties": {"data_size": 14469098, "index_size": 65568, "filter_size": 47228, "raw_key_size": 60920, "raw_average_key_size": 20, "raw_value_size": 14380621, "raw_average_value_size": 4856, "num_data_blocks": 1906, "num_entries": 2961, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:16:28.800196 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1383] Compacted 1@0 + 1@1 files to L1 => 14582821 bytes
>2018-07-12 01:16:28.842057 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.841986) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.1 rd, 67.4 wr, level 1, files in(1, 1) out(1) MB in(2.4, 13.3) out(13.9), read-write-amplify(12.5) write-amplify(5.9) OK, records in: 3469, records dropped: 508
>
>2018-07-12 01:16:28.842066 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:16:28.842007) EVENT_LOG_v1 {"time_micros": 1531358188841998, "job": 1383, "event": "compaction_finished", "compaction_time_micros": 216476, "output_level": 1, "num_output_files": 1, "total_output_size": 14582821, "num_input_records": 3469, "num_output_records": 2961, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:16:28.842594 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188842590, "job": 1383, "event": "table_file_deletion", "file_number": 11201}
>2018-07-12 01:16:28.845328 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358188845325, "job": 1383, "event": "table_file_deletion", "file_number": 11199}
>2018-07-12 01:17:12.362068 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:18:12.392458 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:19:01.480842 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11203. Immutable memtables: 0.
>
>2018-07-12 01:19:01.480887 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1384] Syncing log #11200
>2018-07-12 01:19:01.497387 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.480878) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:19:01.497398 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1384] Flushing memtable with next log file: 11203
>
>2018-07-12 01:19:01.497412 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341497404, "job": 1384, "event": "flush_started", "num_memtables": 1, "num_entries": 787, "num_deletes": 250, "memory_usage": 1152312}
>2018-07-12 01:19:01.497414 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1384] Level-0 flush table #11204: started
>2018-07-12 01:19:01.539595 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341539581, "cf_name": "default", "job": 1384, "event": "table_file_creation", "file_number": 11204, "file_size": 622199, "table_properties": {"data_size": 615979, "index_size": 2707, "filter_size": 2594, "raw_key_size": 7330, "raw_average_key_size": 21, "raw_value_size": 609424, "raw_average_value_size": 1797, "num_data_blocks": 85, "num_entries": 339, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:19:01.539605 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1384] Level-0 flush table #11204: 622199 bytes OK
>2018-07-12 01:19:01.589673 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.556102) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11204 started
>2018-07-12 01:19:01.589681 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.589637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11204: memtable #1 done
>2018-07-12 01:19:01.589684 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.589652) EVENT_LOG_v1 {"time_micros": 1531358341589645, "job": 1384, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:19:01.589686 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.589663) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:19:01.589694 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1384] Try to delete WAL files size 1135775, prev total WAL file size 1135775, number of live WAL files 2.
>
>2018-07-12 01:19:01.590079 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:19:01.590131 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.590124) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:19:01.590140 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1385] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:19:01.590146 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1384 Base level 0, inputs: [11204(607KB)], [11202(13MB)]
>
>2018-07-12 01:19:01.590159 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341590149, "job": 1385, "event": "compaction_started", "files_L0": [11204], "files_L1": [11202], "score": -1, "input_data_size": 15205020}
>2018-07-12 01:19:01.757025 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1385] Generated table #11205: 2864 keys, 13979397 bytes
>2018-07-12 01:19:01.757069 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341757042, "cf_name": "default", "job": 1385, "event": "table_file_creation", "file_number": 11205, "file_size": 13979397, "table_properties": {"data_size": 13871254, "index_size": 62029, "filter_size": 45187, "raw_key_size": 58632, "raw_average_key_size": 20, "raw_value_size": 13786714, "raw_average_value_size": 4813, "num_data_blocks": 1809, "num_entries": 2864, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:19:01.790212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1385] Compacted 1@0 + 1@1 files to L1 => 13979397 bytes
>2018-07-12 01:19:01.823755 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.823688) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.0 rd, 69.9 wr, level 1, files in(1, 1) out(1) MB in(0.6, 13.9) out(13.3), read-write-amplify(46.9) write-amplify(22.5) OK, records in: 3300, records dropped: 436
>
>2018-07-12 01:19:01.823764 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:19:01.823709) EVENT_LOG_v1 {"time_micros": 1531358341823700, "job": 1385, "event": "compaction_finished", "compaction_time_micros": 200037, "output_level": 1, "num_output_files": 1, "total_output_size": 13979397, "num_input_records": 3300, "num_output_records": 2864, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:19:01.823942 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341823937, "job": 1385, "event": "table_file_deletion", "file_number": 11204}
>2018-07-12 01:19:01.826633 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358341826630, "job": 1385, "event": "table_file_deletion", "file_number": 11202}
>2018-07-12 01:19:01.826685 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:19:12.416580 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:20:07.571953 7f29149e4700  1 mon.magna102@0(leader).log v22277 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22277)
>2018-07-12 01:20:12.447352 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:20:17.572908 7f29149e4700  1 mon.magna102@0(leader).log v22278 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22278)
>2018-07-12 01:21:12.469816 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:22:12.500194 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:23:12.522174 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:24:12.551308 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:24:40.663226 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11206. Immutable memtables: 0.
>
>2018-07-12 01:24:40.663276 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1386] Syncing log #11203
>2018-07-12 01:24:40.679890 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.663266) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:24:40.679901 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1386] Flushing memtable with next log file: 11206
>
>2018-07-12 01:24:40.679915 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358680679908, "job": 1386, "event": "flush_started", "num_memtables": 1, "num_entries": 1469, "num_deletes": 251, "memory_usage": 2588392}
>2018-07-12 01:24:40.679919 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1386] Level-0 flush table #11207: started
>2018-07-12 01:24:40.738783 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358680738768, "cf_name": "default", "job": 1386, "event": "table_file_creation", "file_number": 11207, "file_size": 2469194, "table_properties": {"data_size": 2448320, "index_size": 11410, "filter_size": 8540, "raw_key_size": 12599, "raw_average_key_size": 20, "raw_value_size": 2431618, "raw_average_value_size": 3986, "num_data_blocks": 346, "num_entries": 610, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:24:40.738794 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1386] Level-0 flush table #11207: 2469194 bytes OK
>2018-07-12 01:24:40.797165 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.755332) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11207 started
>2018-07-12 01:24:40.797173 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.797128) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11207: memtable #1 done
>2018-07-12 01:24:40.797176 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.797142) EVENT_LOG_v1 {"time_micros": 1531358680797136, "job": 1386, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:24:40.797185 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.797155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:24:40.797192 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1386] Try to delete WAL files size 2558387, prev total WAL file size 2558387, number of live WAL files 2.
>
>2018-07-12 01:24:40.797800 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:24:40.797979 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 01:24:40.797986 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 120707.7 total, 839.4 interval
>Cumulative writes: 147K writes, 690K keys, 147K commit groups, 1.0 writes per commit group, ingest: 1.12 GB, 0.01 MB/s
>Cumulative WAL: 147K writes, 147K syncs, 1.00 writes per sync, written: 1.12 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 852 writes, 3730 keys, 852 commit groups, 1.0 writes per commit group, ingest: 5.99 MB, 0.01 MB/s
>Interval WAL: 852 writes, 852 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.35 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       694    0.077       0      0
>  L1      1/1   13.33 MB   0.0     10.9     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       154       691    0.223   2484K   303K
> Sum      2/2   15.69 MB   0.0     10.9     0.9     10.0      10.9      0.9       0.0  11.8     53.9     53.9       208      1385    0.150   2484K   303K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   8.6     53.1     54.5         1         6    0.140     10K   1381
>Uptime(secs): 120707.7 total, 120707.7 interval
>Flush(GB): cumulative 0.931, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.94 GB write, 0.09 MB/s write, 10.94 GB read, 0.09 MB/s read, 207.9 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    2.35 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       694    0.077       0      0
>  L1      1/1   13.33 MB   0.0     10.9     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       154       691    0.223   2484K   303K
> Sum      2/2   15.69 MB   0.0     10.9     0.9     10.0      10.9      0.9       0.0  11.8     53.9     53.9       208      1385    0.150   2484K   303K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 120707.7 total, 120707.7 interval
>Flush(GB): cumulative 0.931, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.94 GB write, 0.09 MB/s write, 10.94 GB read, 0.09 MB/s read, 207.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 01:24:40.798006 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:24:40.798000) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:24:40.798011 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1387] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:24:40.798016 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1386 Base level 0, inputs: [11207(2411KB)], [11205(13MB)]
>
>2018-07-12 01:24:40.798028 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358680798019, "job": 1387, "event": "compaction_started", "files_L0": [11207], "files_L1": [11205], "score": -1, "input_data_size": 16448591}
>2018-07-12 01:24:40.972859 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1387] Generated table #11208: 2962 keys, 14561882 bytes
>2018-07-12 01:24:40.972903 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358680972876, "cf_name": "default", "job": 1387, "event": "table_file_creation", "file_number": 11208, "file_size": 14561882, "table_properties": {"data_size": 14448319, "index_size": 65467, "filter_size": 47169, "raw_key_size": 60930, "raw_average_key_size": 20, "raw_value_size": 14359905, "raw_average_value_size": 4848, "num_data_blocks": 1903, "num_entries": 2962, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:24:41.014389 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1387] Compacted 1@0 + 1@1 files to L1 => 14561882 bytes
>2018-07-12 01:24:41.047921 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:24:41.047852) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.0 rd, 67.3 wr, level 1, files in(1, 1) out(1) MB in(2.4, 13.3) out(13.9), read-write-amplify(12.6) write-amplify(5.9) OK, records in: 3474, records dropped: 512
>
>2018-07-12 01:24:41.047930 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:24:41.047873) EVENT_LOG_v1 {"time_micros": 1531358681047864, "job": 1387, "event": "compaction_finished", "compaction_time_micros": 216345, "output_level": 1, "num_output_files": 1, "total_output_size": 14561882, "num_input_records": 3474, "num_output_records": 2962, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:24:41.048450 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358681048445, "job": 1387, "event": "table_file_deletion", "file_number": 11207}
>2018-07-12 01:24:41.051141 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358681051138, "job": 1387, "event": "table_file_deletion", "file_number": 11205}
>2018-07-12 01:25:07.766944 7f29149e4700  1 mon.magna102@0(leader).log v22279 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22279)
>2018-07-12 01:25:12.583947 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:25:20.546437 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:25:20.694647 7f29149e4700  1 mon.magna102@0(leader).log v22280 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22280)
>2018-07-12 01:25:26.586327 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:25:26.703404 7f29149e4700  1 mon.magna102@0(leader).log v22281 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22281)
>2018-07-12 01:26:12.606426 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:26:26.576140 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:26:26.750642 7f29149e4700  1 mon.magna102@0(leader).log v22282 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22282)
>2018-07-12 01:26:36.580604 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:26:36.751489 7f29149e4700  1 mon.magna102@0(leader).log v22283 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22283)
>2018-07-12 01:26:52.587577 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:26:52.736082 7f29149e4700  1 mon.magna102@0(leader).log v22284 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22284)
>2018-07-12 01:27:01.627088 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:27:01.728471 7f29149e4700  1 mon.magna102@0(leader).log v22285 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22285)
>2018-07-12 01:27:12.629166 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:27:21.696964 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11209. Immutable memtables: 0.
>
>2018-07-12 01:27:21.697008 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1388] Syncing log #11206
>2018-07-12 01:27:21.713320 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.696998) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:27:21.713332 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1388] Flushing memtable with next log file: 11209
>
>2018-07-12 01:27:21.713348 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358841713339, "job": 1388, "event": "flush_started", "num_memtables": 1, "num_entries": 871, "num_deletes": 250, "memory_usage": 1356376}
>2018-07-12 01:27:21.713352 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1388] Level-0 flush table #11210: started
>2018-07-12 01:27:21.764005 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358841763991, "cf_name": "default", "job": 1388, "event": "table_file_creation", "file_number": 11210, "file_size": 781588, "table_properties": {"data_size": 774325, "index_size": 3265, "filter_size": 3079, "raw_key_size": 7888, "raw_average_key_size": 21, "raw_value_size": 766968, "raw_average_value_size": 2095, "num_data_blocks": 102, "num_entries": 366, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:27:21.764016 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1388] Level-0 flush table #11210: 781588 bytes OK
>2018-07-12 01:27:21.822369 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.780536) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11210 started
>2018-07-12 01:27:21.822377 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.822333) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11210: memtable #1 done
>2018-07-12 01:27:21.822379 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.822348) EVENT_LOG_v1 {"time_micros": 1531358841822341, "job": 1388, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:27:21.822382 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.822359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:27:21.822401 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1388] Try to delete WAL files size 1338372, prev total WAL file size 1338372, number of live WAL files 2.
>
>2018-07-12 01:27:21.822820 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:27:21.822884 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:27:21.822877) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:27:21.822891 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1389] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:27:21.822897 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1388 Base level 0, inputs: [11210(763KB)], [11208(13MB)]
>
>2018-07-12 01:27:21.822910 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358841822901, "job": 1389, "event": "compaction_started", "files_L0": [11210], "files_L1": [11208], "score": -1, "input_data_size": 15343470}
>2018-07-12 01:27:21.998062 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1389] Generated table #11211: 2894 keys, 14144881 bytes
>2018-07-12 01:27:21.998106 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358841998079, "cf_name": "default", "job": 1389, "event": "table_file_creation", "file_number": 11211, "file_size": 14144881, "table_properties": {"data_size": 14035567, "index_size": 62679, "filter_size": 45708, "raw_key_size": 59237, "raw_average_key_size": 20, "raw_value_size": 13950146, "raw_average_value_size": 4820, "num_data_blocks": 1828, "num_entries": 2894, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:27:22.040816 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1389] Compacted 1@0 + 1@1 files to L1 => 14144881 bytes
>2018-07-12 01:27:22.074344 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:27:22.074276) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 70.4 rd, 64.9 wr, level 1, files in(1, 1) out(1) MB in(0.7, 13.9) out(13.5), read-write-amplify(37.7) write-amplify(18.1) OK, records in: 3328, records dropped: 434
>
>2018-07-12 01:27:22.074353 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:27:22.074298) EVENT_LOG_v1 {"time_micros": 1531358842074288, "job": 1389, "event": "compaction_finished", "compaction_time_micros": 217889, "output_level": 1, "num_output_files": 1, "total_output_size": 14144881, "num_input_records": 3328, "num_output_records": 2894, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:27:22.074591 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358842074587, "job": 1389, "event": "table_file_deletion", "file_number": 11210}
>2018-07-12 01:27:22.077363 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531358842077360, "job": 1389, "event": "table_file_deletion", "file_number": 11208}
>2018-07-12 01:27:22.077479 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:28:02.619349 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:28:02.769572 7f29149e4700  1 mon.magna102@0(leader).log v22286 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22286)
>2018-07-12 01:28:10.622918 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:28:10.770357 7f29149e4700  1 mon.magna102@0(leader).log v22287 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22287)
>2018-07-12 01:28:12.661998 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:28:30.631903 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:28:30.789957 7f29149e4700  1 mon.magna102@0(leader).log v22288 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22288)
>2018-07-12 01:28:36.673648 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:28:36.790700 7f29149e4700  1 mon.magna102@0(leader).log v22289 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22289)
>2018-07-12 01:29:12.693112 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:29:36.661727 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:29:36.812973 7f29149e4700  1 mon.magna102@0(leader).log v22290 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22290)
>2018-07-12 01:29:46.666268 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:29:46.813810 7f29149e4700  1 mon.magna102@0(leader).log v22291 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22291)
>2018-07-12 01:30:06.675306 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:30:06.832346 7f29149e4700  1 mon.magna102@0(leader).log v22292 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22292)
>2018-07-12 01:30:12.715962 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:31:12.704833 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:31:12.738200 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:31:12.855241 7f29149e4700  1 mon.magna102@0(leader).log v22293 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22293)
>2018-07-12 01:31:20.708312 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:31:20.855969 7f29149e4700  1 mon.magna102@0(leader).log v22294 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22294)
>2018-07-12 01:31:36.715467 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:31:36.865533 7f29149e4700  1 mon.magna102@0(leader).log v22295 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22295)
>2018-07-12 01:31:41.748780 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:31:41.849329 7f29149e4700  1 mon.magna102@0(leader).log v22296 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22296)
>2018-07-12 01:32:12.769293 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:32:22.887228 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11212. Immutable memtables: 0.
>
>2018-07-12 01:32:22.887273 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1390] Syncing log #11209
>2018-07-12 01:32:22.903733 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:32:22.887264) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:32:22.903744 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1390] Flushing memtable with next log file: 11212
>
>2018-07-12 01:32:22.903758 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359142903751, "job": 1390, "event": "flush_started", "num_memtables": 1, "num_entries": 1401, "num_deletes": 251, "memory_usage": 2515384}
>2018-07-12 01:32:22.903761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1390] Level-0 flush table #11213: started
>2018-07-12 01:32:22.962631 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359142962616, "cf_name": "default", "job": 1390, "event": "table_file_creation", "file_number": 11213, "file_size": 2387433, "table_properties": {"data_size": 2367547, "index_size": 10747, "filter_size": 8215, "raw_key_size": 12260, "raw_average_key_size": 20, "raw_value_size": 2351561, "raw_average_value_size": 3952, "num_data_blocks": 326, "num_entries": 595, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:32:22.962642 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1390] Level-0 flush table #11213: 2387433 bytes OK
>2018-07-12 01:32:23.012691 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:32:22.979197) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11213 started
>2018-07-12 01:32:23.012699 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.012653) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11213: memtable #1 done
>2018-07-12 01:32:23.012701 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.012667) EVENT_LOG_v1 {"time_micros": 1531359143012661, "job": 1390, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:32:23.012703 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.012680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:32:23.012710 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1390] Try to delete WAL files size 2487580, prev total WAL file size 2487580, number of live WAL files 2.
>
>2018-07-12 01:32:23.013325 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:32:23.013379 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.013371) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:32:23.013388 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1391] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:32:23.013394 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1390 Base level 0, inputs: [11213(2331KB)], [11211(13MB)]
>
>2018-07-12 01:32:23.013407 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359143013397, "job": 1391, "event": "compaction_started", "files_L0": [11213], "files_L1": [11211], "score": -1, "input_data_size": 16532314}
>2018-07-12 01:32:23.188378 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1391] Generated table #11214: 2979 keys, 14652185 bytes
>2018-07-12 01:32:23.188409 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359143188395, "cf_name": "default", "job": 1391, "event": "table_file_creation", "file_number": 11214, "file_size": 14652185, "table_properties": {"data_size": 14538398, "index_size": 65478, "filter_size": 47382, "raw_key_size": 61250, "raw_average_key_size": 20, "raw_value_size": 14449738, "raw_average_value_size": 4850, "num_data_blocks": 1903, "num_entries": 2979, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:32:23.221577 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1391] Compacted 1@0 + 1@1 files to L1 => 14652185 bytes
>2018-07-12 01:32:23.255087 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.255018) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.4 rd, 70.4 wr, level 1, files in(1, 1) out(1) MB in(2.3, 13.5) out(14.0), read-write-amplify(13.1) write-amplify(6.1) OK, records in: 3489, records dropped: 510
>
>2018-07-12 01:32:23.255099 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:32:23.255040) EVENT_LOG_v1 {"time_micros": 1531359143255030, "job": 1391, "event": "compaction_finished", "compaction_time_micros": 208154, "output_level": 1, "num_output_files": 1, "total_output_size": 14652185, "num_input_records": 3489, "num_output_records": 2979, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:32:23.255622 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359143255618, "job": 1391, "event": "table_file_deletion", "file_number": 11213}
>2018-07-12 01:32:23.258399 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359143258396, "job": 1391, "event": "table_file_deletion", "file_number": 11211}
>2018-07-12 01:32:46.746716 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:32:46.925483 7f29149e4700  1 mon.magna102@0(leader).log v22297 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22297)
>2018-07-12 01:32:56.751178 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:32:56.909663 7f29149e4700  1 mon.magna102@0(leader).log v22298 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22298)
>2018-07-12 01:33:12.758116 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:33:12.793919 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:33:12.910956 7f29149e4700  1 mon.magna102@0(leader).log v22299 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22299)
>2018-07-12 01:33:21.794598 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:33:21.903352 7f29149e4700  1 mon.magna102@0(leader).log v22300 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22300)
>2018-07-12 01:34:12.824634 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:34:22.789302 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:34:22.942455 7f29149e4700  1 mon.magna102@0(leader).log v22301 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22301)
>2018-07-12 01:34:30.792946 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:34:30.968159 7f29149e4700  1 mon.magna102@0(leader).log v22302 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22302)
>2018-07-12 01:34:46.800017 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:34:46.977759 7f29149e4700  1 mon.magna102@0(leader).log v22303 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22303)
>2018-07-12 01:34:51.844322 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:34:51.969771 7f29149e4700  1 mon.magna102@0(leader).log v22304 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22304)
>2018-07-12 01:35:12.847492 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:35:41.933278 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11215. Immutable memtables: 0.
>
>2018-07-12 01:35:41.933322 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1392] Syncing log #11212
>2018-07-12 01:35:41.949777 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:35:41.933313) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:35:41.949788 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1392] Flushing memtable with next log file: 11215
>
>2018-07-12 01:35:41.949802 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359341949795, "job": 1392, "event": "flush_started", "num_memtables": 1, "num_entries": 1012, "num_deletes": 250, "memory_usage": 1670792}
>2018-07-12 01:35:41.949805 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1392] Level-0 flush table #11216: started
>2018-07-12 01:35:42.000413 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359342000398, "cf_name": "default", "job": 1392, "event": "table_file_creation", "file_number": 11216, "file_size": 947195, "table_properties": {"data_size": 938749, "index_size": 3937, "filter_size": 3590, "raw_key_size": 8329, "raw_average_key_size": 21, "raw_value_size": 930595, "raw_average_value_size": 2398, "num_data_blocks": 123, "num_entries": 388, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:35:42.000424 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1392] Level-0 flush table #11216: 947195 bytes OK
>2018-07-12 01:35:42.058678 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.016939) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11216 started
>2018-07-12 01:35:42.058685 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.058641) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11216: memtable #1 done
>2018-07-12 01:35:42.058688 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.058656) EVENT_LOG_v1 {"time_micros": 1531359342058650, "job": 1392, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:35:42.058690 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.058668) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:35:42.058697 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1392] Try to delete WAL files size 1650356, prev total WAL file size 1650356, number of live WAL files 2.
>
>2018-07-12 01:35:42.059149 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:35:42.059342 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 01:35:42.059349 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 121368.9 total, 661.3 interval
>Cumulative writes: 148K writes, 693K keys, 148K commit groups, 1.0 writes per commit group, ingest: 1.13 GB, 0.01 MB/s
>Cumulative WAL: 148K writes, 148K syncs, 1.00 writes per sync, written: 1.13 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 718 writes, 3284 keys, 718 commit groups, 1.0 writes per commit group, ingest: 5.22 MB, 0.01 MB/s
>Interval WAL: 718 writes, 718 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   925.00 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       697    0.077       0      0
>  L1      1/1   13.97 MB   0.0     11.0     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       155       694    0.223   2494K   304K
> Sum      2/2   14.88 MB   0.0     11.0     0.9     10.0      11.0      0.9       0.0  11.8     53.9     53.9       209      1391    0.150   2494K   304K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.5     54.1     53.1         1         6    0.142     10K   1456
>Uptime(secs): 121368.9 total, 121368.9 interval
>Flush(GB): cumulative 0.934, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.98 GB write, 0.09 MB/s write, 10.98 GB read, 0.09 MB/s read, 208.8 seconds
>Interval compaction: 0.04 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   925.00 KB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       697    0.077       0      0
>  L1      1/1   13.97 MB   0.0     11.0     0.9     10.0      10.0     -0.0       0.0  10.8     72.6     66.4       155       694    0.223   2494K   304K
> Sum      2/2   14.88 MB   0.0     11.0     0.9     10.0      11.0      0.9       0.0  11.8     53.9     53.9       209      1391    0.150   2494K   304K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 121368.9 total, 121368.9 interval
>Flush(GB): cumulative 0.934, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 10.98 GB write, 0.09 MB/s write, 10.98 GB read, 0.09 MB/s read, 208.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 01:35:42.059369 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.059362) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:35:42.059373 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1393] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:35:42.059378 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1392 Base level 0, inputs: [11216(924KB)], [11214(13MB)]
>
>2018-07-12 01:35:42.059390 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359342059382, "job": 1393, "event": "compaction_started", "files_L0": [11216], "files_L1": [11214], "score": -1, "input_data_size": 15599380}
>2018-07-12 01:35:42.234402 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1393] Generated table #11217: 2952 keys, 14517825 bytes
>2018-07-12 01:35:42.234439 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359342234424, "cf_name": "default", "job": 1393, "event": "table_file_creation", "file_number": 11217, "file_size": 14517825, "table_properties": {"data_size": 14406019, "index_size": 64048, "filter_size": 46831, "raw_key_size": 60416, "raw_average_key_size": 20, "raw_value_size": 14318813, "raw_average_value_size": 4850, "num_data_blocks": 1868, "num_entries": 2952, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:35:42.267590 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1393] Compacted 1@0 + 1@1 files to L1 => 14517825 bytes
>2018-07-12 01:35:42.309430 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.309359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 74.9 rd, 69.7 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.0) out(13.8), read-write-amplify(31.8) write-amplify(15.3) OK, records in: 3367, records dropped: 415
>
>2018-07-12 01:35:42.309439 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:35:42.309381) EVENT_LOG_v1 {"time_micros": 1531359342309371, "job": 1393, "event": "compaction_finished", "compaction_time_micros": 208185, "output_level": 1, "num_output_files": 1, "total_output_size": 14517825, "num_input_records": 3367, "num_output_records": 2952, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:35:42.309673 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359342309669, "job": 1393, "event": "table_file_deletion", "file_number": 11216}
>2018-07-12 01:35:42.312373 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359342312370, "job": 1393, "event": "table_file_deletion", "file_number": 11214}
>2018-07-12 01:35:42.312425 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:35:56.831548 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:35:56.987450 7f29149e4700  1 mon.magna102@0(leader).log v22305 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22305)
>2018-07-12 01:36:04.835101 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:36:04.988155 7f29149e4700  1 mon.magna102@0(leader).log v22306 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22306)
>2018-07-12 01:36:12.871874 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:36:20.842260 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:36:20.997884 7f29149e4700  1 mon.magna102@0(leader).log v22307 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22307)
>2018-07-12 01:36:26.901363 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:36:27.023491 7f29149e4700  1 mon.magna102@0(leader).log v22308 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22308)
>2018-07-12 01:37:12.894527 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:37:32.874367 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:37:33.021261 7f29149e4700  1 mon.magna102@0(leader).log v22309 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22309)
>2018-07-12 01:37:40.878198 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:37:41.038644 7f29149e4700  1 mon.magna102@0(leader).log v22310 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22310)
>2018-07-12 01:37:56.885326 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:37:57.039912 7f29149e4700  1 mon.magna102@0(leader).log v22311 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22311)
>2018-07-12 01:38:01.927988 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:38:02.023699 7f29149e4700  1 mon.magna102@0(leader).log v22312 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22312)
>2018-07-12 01:38:12.934049 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:39:12.919226 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:39:12.955097 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8797 MB, avail 861 GB
>2018-07-12 01:39:13.072112 7f29149e4700  1 mon.magna102@0(leader).log v22313 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22313)
>2018-07-12 01:39:14.920205 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:39:15.073649 7f29149e4700  1 mon.magna102@0(leader).log v22314 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22314)
>2018-07-12 01:39:32.928066 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:39:33.083387 7f29149e4700  1 mon.magna102@0(leader).log v22315 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22315)
>2018-07-12 01:39:41.958651 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:39:42.067441 7f29149e4700  1 mon.magna102@0(leader).log v22316 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22316)
>2018-07-12 01:40:01.077528 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11218. Immutable memtables: 0.
>
>2018-07-12 01:40:01.077571 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1394] Syncing log #11215
>2018-07-12 01:40:01.094121 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.077562) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:40:01.094132 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1394] Flushing memtable with next log file: 11218
>
>2018-07-12 01:40:01.094146 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601094139, "job": 1394, "event": "flush_started", "num_memtables": 1, "num_entries": 1262, "num_deletes": 251, "memory_usage": 2217352}
>2018-07-12 01:40:01.094149 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1394] Level-0 flush table #11219: started
>2018-07-12 01:40:01.152966 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601152951, "cf_name": "default", "job": 1394, "event": "table_file_creation", "file_number": 11219, "file_size": 2105439, "table_properties": {"data_size": 2087807, "index_size": 9399, "filter_size": 7313, "raw_key_size": 11439, "raw_average_key_size": 20, "raw_value_size": 2073324, "raw_average_value_size": 3729, "num_data_blocks": 286, "num_entries": 556, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:40:01.152977 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1394] Level-0 flush table #11219: 2105439 bytes OK
>2018-07-12 01:40:01.203055 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.169538) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11219 started
>2018-07-12 01:40:01.203063 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.203016) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11219: memtable #1 done
>2018-07-12 01:40:01.203065 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.203031) EVENT_LOG_v1 {"time_micros": 1531359601203025, "job": 1394, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:40:01.203067 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.203044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:40:01.203073 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1394] Try to delete WAL files size 2191447, prev total WAL file size 2191447, number of live WAL files 2.
>
>2018-07-12 01:40:01.203652 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:40:01.203706 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.203699) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:40:01.203715 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1395] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:40:01.203721 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1394 Base level 0, inputs: [11219(2056KB)], [11217(13MB)]
>
>2018-07-12 01:40:01.203734 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601203724, "job": 1395, "event": "compaction_started", "files_L0": [11219], "files_L1": [11217], "score": -1, "input_data_size": 16623264}
>2018-07-12 01:40:01.387074 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1395] Generated table #11220: 2998 keys, 14747669 bytes
>2018-07-12 01:40:01.387117 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601387090, "cf_name": "default", "job": 1395, "event": "table_file_creation", "file_number": 11220, "file_size": 14747669, "table_properties": {"data_size": 14633594, "index_size": 65526, "filter_size": 47622, "raw_key_size": 61608, "raw_average_key_size": 20, "raw_value_size": 14544631, "raw_average_value_size": 4851, "num_data_blocks": 1904, "num_entries": 2998, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:40:01.420268 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1395] Compacted 1@0 + 1@1 files to L1 => 14747669 bytes
>2018-07-12 01:40:01.453800 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.453732) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.8 rd, 68.1 wr, level 1, files in(1, 1) out(1) MB in(2.0, 13.8) out(14.1), read-write-amplify(14.9) write-amplify(7.0) OK, records in: 3508, records dropped: 510
>
>2018-07-12 01:40:01.453809 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:40:01.453754) EVENT_LOG_v1 {"time_micros": 1531359601453744, "job": 1395, "event": "compaction_finished", "compaction_time_micros": 216520, "output_level": 1, "num_output_files": 1, "total_output_size": 14747669, "num_input_records": 3508, "num_output_records": 2998, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:40:01.454298 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601454293, "job": 1395, "event": "table_file_deletion", "file_number": 11219}
>2018-07-12 01:40:01.457084 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359601457081, "job": 1395, "event": "table_file_deletion", "file_number": 11217}
>2018-07-12 01:40:12.981354 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:40:46.961526 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:40:47.109408 7f29149e4700  1 mon.magna102@0(leader).log v22317 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22317)
>2018-07-12 01:40:54.965127 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:40:55.110164 7f29149e4700  1 mon.magna102@0(leader).log v22318 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22318)
>2018-07-12 01:41:06.970385 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:41:07.127820 7f29149e4700  1 mon.magna102@0(leader).log v22319 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22319)
>2018-07-12 01:41:12.010947 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:41:12.104496 7f29149e4700  1 mon.magna102@0(leader).log v22320 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22320)
>2018-07-12 01:41:12.981527 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:42:12.981717 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:42:23.004428 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:42:23.177472 7f29149e4700  1 mon.magna102@0(leader).log v22321 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22321)
>2018-07-12 01:42:31.008234 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:42:31.178151 7f29149e4700  1 mon.magna102@0(leader).log v22322 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22322)
>2018-07-12 01:42:45.014411 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:42:45.162735 7f29149e4700  1 mon.magna102@0(leader).log v22323 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22323)
>2018-07-12 01:42:52.054079 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:42:52.146762 7f29149e4700  1 mon.magna102@0(leader).log v22324 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22324)
>2018-07-12 01:43:12.981896 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:43:53.044894 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:43:53.194051 7f29149e4700  1 mon.magna102@0(leader).log v22325 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22325)
>2018-07-12 01:44:01.048504 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:44:01.203120 7f29149e4700  1 mon.magna102@0(leader).log v22326 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22326)
>2018-07-12 01:44:02.161874 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11221. Immutable memtables: 0.
>
>2018-07-12 01:44:02.161918 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1396] Syncing log #11218
>2018-07-12 01:44:02.178387 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.161909) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:44:02.178398 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1396] Flushing memtable with next log file: 11221
>
>2018-07-12 01:44:02.178412 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842178405, "job": 1396, "event": "flush_started", "num_memtables": 1, "num_entries": 1175, "num_deletes": 250, "memory_usage": 2050896}
>2018-07-12 01:44:02.178415 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1396] Level-0 flush table #11222: started
>2018-07-12 01:44:02.228920 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842228906, "cf_name": "default", "job": 1396, "event": "table_file_creation", "file_number": 11222, "file_size": 1164640, "table_properties": {"data_size": 1154739, "index_size": 4741, "filter_size": 4240, "raw_key_size": 8871, "raw_average_key_size": 21, "raw_value_size": 1145631, "raw_average_value_size": 2760, "num_data_blocks": 148, "num_entries": 415, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:44:02.228930 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1396] Level-0 flush table #11222: 1164640 bytes OK
>2018-07-12 01:44:02.287331 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.245491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11222 started
>2018-07-12 01:44:02.287339 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.287295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11222: memtable #1 done
>2018-07-12 01:44:02.287341 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.287309) EVENT_LOG_v1 {"time_micros": 1531359842287303, "job": 1396, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:44:02.287343 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.287321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:44:02.287350 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1396] Try to delete WAL files size 2027085, prev total WAL file size 2027085, number of live WAL files 2.
>
>2018-07-12 01:44:02.287883 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:44:02.287939 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.287932) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:44:02.287948 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1397] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:44:02.287954 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1396 Base level 0, inputs: [11222(1137KB)], [11220(14MB)]
>
>2018-07-12 01:44:02.287966 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842287957, "job": 1397, "event": "compaction_started", "files_L0": [11222], "files_L1": [11220], "score": -1, "input_data_size": 15912309}
>2018-07-12 01:44:02.471384 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1397] Generated table #11223: 3019 keys, 14974769 bytes
>2018-07-12 01:44:02.471421 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842471405, "cf_name": "default", "job": 1397, "event": "table_file_creation", "file_number": 11223, "file_size": 14974769, "table_properties": {"data_size": 14859965, "index_size": 65690, "filter_size": 48187, "raw_key_size": 61778, "raw_average_key_size": 20, "raw_value_size": 14770656, "raw_average_value_size": 4892, "num_data_blocks": 1916, "num_entries": 3019, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:44:02.504549 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1397] Compacted 1@0 + 1@1 files to L1 => 14974769 bytes
>2018-07-12 01:44:02.546418 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.546349) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.5 rd, 69.1 wr, level 1, files in(1, 1) out(1) MB in(1.1, 14.1) out(14.3), read-write-amplify(26.5) write-amplify(12.9) OK, records in: 3413, records dropped: 394
>
>2018-07-12 01:44:02.546427 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:44:02.546370) EVENT_LOG_v1 {"time_micros": 1531359842546360, "job": 1397, "event": "compaction_finished", "compaction_time_micros": 216566, "output_level": 1, "num_output_files": 1, "total_output_size": 14974769, "num_input_records": 3413, "num_output_records": 3019, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:44:02.546702 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842546698, "job": 1397, "event": "table_file_deletion", "file_number": 11222}
>2018-07-12 01:44:02.549432 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531359842549429, "job": 1397, "event": "table_file_deletion", "file_number": 11220}
>2018-07-12 01:44:02.549485 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:44:12.982077 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:44:17.055945 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:44:17.223201 7f29149e4700  1 mon.magna102@0(leader).log v22327 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22327)
>2018-07-12 01:44:22.128959 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:44:22.241538 7f29149e4700  1 mon.magna102@0(leader).log v22328 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22328)
>2018-07-12 01:45:12.982260 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:45:33.089941 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:45:33.239704 7f29149e4700  1 mon.magna102@0(leader).log v22329 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22329)
>2018-07-12 01:45:41.093659 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:45:41.248762 7f29149e4700  1 mon.magna102@0(leader).log v22330 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22330)
>2018-07-12 01:45:53.098862 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:45:53.266439 7f29149e4700  1 mon.magna102@0(leader).log v22331 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22331)
>2018-07-12 01:46:02.141709 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:46:02.258798 7f29149e4700  1 mon.magna102@0(leader).log v22332 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22332)
>2018-07-12 01:46:12.982445 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:47:07.132109 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:47:07.281566 7f29149e4700  1 mon.magna102@0(leader).log v22333 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22333)
>2018-07-12 01:47:12.982615 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:47:15.135617 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:47:15.290595 7f29149e4700  1 mon.magna102@0(leader).log v22334 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22334)
>2018-07-12 01:47:31.142882 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:47:31.291883 7f29149e4700  1 mon.magna102@0(leader).log v22335 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22335)
>2018-07-12 01:47:37.183849 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:47:37.300840 7f29149e4700  1 mon.magna102@0(leader).log v22336 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22336)
>2018-07-12 01:47:39.284715 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11224. Immutable memtables: 0.
>
>2018-07-12 01:47:39.284759 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1398] Syncing log #11221
>2018-07-12 01:47:39.301268 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.284749) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:47:39.301280 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1398] Flushing memtable with next log file: 11224
>
>2018-07-12 01:47:39.301307 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059301299, "job": 1398, "event": "flush_started", "num_memtables": 1, "num_entries": 1099, "num_deletes": 251, "memory_usage": 1891504}
>2018-07-12 01:47:39.301310 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1398] Level-0 flush table #11225: started
>2018-07-12 01:47:39.353027 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059353012, "cf_name": "default", "job": 1398, "event": "table_file_creation", "file_number": 11225, "file_size": 1797224, "table_properties": {"data_size": 1782122, "index_size": 7884, "filter_size": 6298, "raw_key_size": 10435, "raw_average_key_size": 20, "raw_value_size": 1769419, "raw_average_value_size": 3483, "num_data_blocks": 240, "num_entries": 508, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:47:39.353037 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1398] Level-0 flush table #11225: 1797224 bytes OK
>2018-07-12 01:47:39.411413 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.369543) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11225 started
>2018-07-12 01:47:39.411421 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.411376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11225: memtable #1 done
>2018-07-12 01:47:39.411424 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.411390) EVENT_LOG_v1 {"time_micros": 1531360059411384, "job": 1398, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:47:39.411426 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.411403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:47:39.411446 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1398] Try to delete WAL files size 1868770, prev total WAL file size 1868770, number of live WAL files 2.
>
>2018-07-12 01:47:39.411965 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:47:39.412164 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 01:47:39.412170 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 122086.3 total, 717.4 interval
>Cumulative writes: 149K writes, 696K keys, 149K commit groups, 1.0 writes per commit group, ingest: 1.13 GB, 0.01 MB/s
>Cumulative WAL: 149K writes, 149K syncs, 1.00 writes per sync, written: 1.13 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 788 writes, 3536 keys, 788 commit groups, 1.0 writes per commit group, ingest: 5.81 MB, 0.01 MB/s
>Interval WAL: 788 writes, 788 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       700    0.077       0      0
>  L1      1/1   14.28 MB   0.0     11.0     0.9     10.1      10.1     -0.0       0.0  10.8     72.5     66.4       156       697    0.223   2505K   306K
> Sum      2/2   16.00 MB   0.0     11.0     0.9     10.1      11.0      0.9       0.0  11.7     53.9     53.9       210      1397    0.150   2505K   306K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   9.7     53.9     55.2         1         6    0.142     10K   1319
>Uptime(secs): 122086.3 total, 122086.3 interval
>Flush(GB): cumulative 0.939, interval 0.005
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.03 GB write, 0.09 MB/s write, 11.03 GB read, 0.09 MB/s read, 209.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.04 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.71 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       700    0.077       0      0
>  L1      1/1   14.28 MB   0.0     11.0     0.9     10.1      10.1     -0.0       0.0  10.8     72.5     66.4       156       697    0.223   2505K   306K
> Sum      2/2   16.00 MB   0.0     11.0     0.9     10.1      11.0      0.9       0.0  11.7     53.9     53.9       210      1397    0.150   2505K   306K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 122086.3 total, 122086.3 interval
>Flush(GB): cumulative 0.939, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.03 GB write, 0.09 MB/s write, 11.03 GB read, 0.09 MB/s read, 209.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 01:47:39.412203 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.412197) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:47:39.412209 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1399] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:47:39.412214 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1398 Base level 0, inputs: [11225(1755KB)], [11223(14MB)]
>
>2018-07-12 01:47:39.412227 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059412217, "job": 1399, "event": "compaction_started", "files_L0": [11225], "files_L1": [11223], "score": -1, "input_data_size": 16771993}
>2018-07-12 01:47:39.587103 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1399] Generated table #11226: 3017 keys, 14830231 bytes
>2018-07-12 01:47:39.587147 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059587120, "cf_name": "default", "job": 1399, "event": "table_file_creation", "file_number": 11226, "file_size": 14830231, "table_properties": {"data_size": 14715937, "index_size": 65543, "filter_size": 47824, "raw_key_size": 61966, "raw_average_key_size": 20, "raw_value_size": 14626696, "raw_average_value_size": 4848, "num_data_blocks": 1904, "num_entries": 3017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:47:39.620296 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1399] Compacted 1@0 + 1@1 files to L1 => 14830231 bytes
>2018-07-12 01:47:39.662145 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.662079) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.6 rd, 71.3 wr, level 1, files in(1, 1) out(1) MB in(1.7, 14.3) out(14.1), read-write-amplify(17.6) write-amplify(8.3) OK, records in: 3527, records dropped: 510
>
>2018-07-12 01:47:39.662153 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:47:39.662100) EVENT_LOG_v1 {"time_micros": 1531360059662091, "job": 1399, "event": "compaction_finished", "compaction_time_micros": 208053, "output_level": 1, "num_output_files": 1, "total_output_size": 14830231, "num_input_records": 3527, "num_output_records": 3017, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:47:39.662605 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059662601, "job": 1399, "event": "table_file_deletion", "file_number": 11225}
>2018-07-12 01:47:39.665497 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360059665494, "job": 1399, "event": "table_file_deletion", "file_number": 11223}
>2018-07-12 01:48:12.982792 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:48:43.175460 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:48:43.326897 7f29149e4700  1 mon.magna102@0(leader).log v22337 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22337)
>2018-07-12 01:48:51.179191 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:48:51.327598 7f29149e4700  1 mon.magna102@0(leader).log v22338 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22338)
>2018-07-12 01:49:05.185440 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:49:05.337300 7f29149e4700  1 mon.magna102@0(leader).log v22339 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22339)
>2018-07-12 01:49:12.220798 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:49:12.346210 7f29149e4700  1 mon.magna102@0(leader).log v22340 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22340)
>2018-07-12 01:49:12.982973 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:50:12.983163 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:50:17.218055 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:50:17.369143 7f29149e4700  1 mon.magna102@0(leader).log v22341 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22341)
>2018-07-12 01:50:25.221546 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:50:25.394871 7f29149e4700  1 mon.magna102@0(leader).log v22342 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22342)
>2018-07-12 01:50:41.228630 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:50:41.379450 7f29149e4700  1 mon.magna102@0(leader).log v22343 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22343)
>2018-07-12 01:50:47.270524 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:50:47.388533 7f29149e4700  1 mon.magna102@0(leader).log v22344 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22344)
>2018-07-12 01:51:12.983352 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:51:53.261015 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:51:53.427838 7f29149e4700  1 mon.magna102@0(leader).log v22345 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22345)
>2018-07-12 01:52:01.264582 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:52:01.429788 7f29149e4700  1 mon.magna102@0(leader).log v22346 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22346)
>2018-07-12 01:52:12.983520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:52:15.270838 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:52:15.414259 7f29149e4700  1 mon.magna102@0(leader).log v22347 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22347)
>2018-07-12 01:52:22.328215 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:52:22.398397 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11227. Immutable memtables: 0.
>
>2018-07-12 01:52:22.398445 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1400] Syncing log #11224
>2018-07-12 01:52:22.414930 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.398435) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:52:22.414942 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1400] Flushing memtable with next log file: 11227
>
>2018-07-12 01:52:22.414956 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342414948, "job": 1400, "event": "flush_started", "num_memtables": 1, "num_entries": 1330, "num_deletes": 250, "memory_usage": 2405648}
>2018-07-12 01:52:22.414959 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1400] Level-0 flush table #11228: started
>2018-07-12 01:52:22.490647 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342490632, "cf_name": "default", "job": 1400, "event": "table_file_creation", "file_number": 11228, "file_size": 1359666, "table_properties": {"data_size": 1348437, "index_size": 5479, "filter_size": 4830, "raw_key_size": 9352, "raw_average_key_size": 21, "raw_value_size": 1338458, "raw_average_value_size": 3048, "num_data_blocks": 171, "num_entries": 439, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 01:52:22.490657 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1400] Level-0 flush table #11228: 1359666 bytes OK
>2018-07-12 01:52:22.524059 7f29149e4700  1 mon.magna102@0(leader).log v22348 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22348)
>2018-07-12 01:52:22.565646 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.523819) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11228 started
>2018-07-12 01:52:22.565655 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.565609) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11228: memtable #1 done
>2018-07-12 01:52:22.565657 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.565624) EVENT_LOG_v1 {"time_micros": 1531360342565618, "job": 1400, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:52:22.565659 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.565636) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:52:22.565679 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1400] Try to delete WAL files size 2379360, prev total WAL file size 2399430, number of live WAL files 2.
>
>2018-07-12 01:52:22.566319 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:52:22.566373 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.566366) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:52:22.566383 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1401] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:52:22.566389 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1400 Base level 0, inputs: [11228(1327KB)], [11226(14MB)]
>
>2018-07-12 01:52:22.566402 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342566392, "job": 1401, "event": "compaction_started", "files_L0": [11228], "files_L1": [11226], "score": -1, "input_data_size": 16189897}
>2018-07-12 01:52:22.749700 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1401] Generated table #11229: 3083 keys, 15396296 bytes
>2018-07-12 01:52:22.749750 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342749722, "cf_name": "default", "job": 1401, "event": "table_file_creation", "file_number": 11229, "file_size": 15396296, "table_properties": {"data_size": 15278724, "index_size": 67197, "filter_size": 49448, "raw_key_size": 63079, "raw_average_key_size": 20, "raw_value_size": 15187454, "raw_average_value_size": 4926, "num_data_blocks": 1960, "num_entries": 3083, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:52:22.791210 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1401] Compacted 1@0 + 1@1 files to L1 => 15396296 bytes
>2018-07-12 01:52:22.833079 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.833007) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 68.5 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.1) out(14.7), read-write-amplify(23.2) write-amplify(11.3) OK, records in: 3456, records dropped: 373
>
>2018-07-12 01:52:22.833088 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.833028) EVENT_LOG_v1 {"time_micros": 1531360342833019, "job": 1401, "event": "compaction_finished", "compaction_time_micros": 224791, "output_level": 1, "num_output_files": 1, "total_output_size": 15396296, "num_input_records": 3456, "num_output_records": 3083, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:52:22.833407 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342833403, "job": 1401, "event": "table_file_deletion", "file_number": 11228}
>2018-07-12 01:52:22.836103 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342836100, "job": 1401, "event": "table_file_deletion", "file_number": 11226}
>2018-07-12 01:52:22.836365 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11230. Immutable memtables: 0.
>
>2018-07-12 01:52:22.836404 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1402] Syncing log #11227
>2018-07-12 01:52:22.850086 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.836396) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:52:22.850097 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1402] Flushing memtable with next log file: 11230
>
>2018-07-12 01:52:22.850110 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342850103, "job": 1402, "event": "flush_started", "num_memtables": 1, "num_entries": 8, "num_deletes": 0, "memory_usage": 22408}
>2018-07-12 01:52:22.850113 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1402] Level-0 flush table #11231: started
>2018-07-12 01:52:22.892243 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342892231, "cf_name": "default", "job": 1402, "event": "table_file_creation", "file_number": 11231, "file_size": 21189, "table_properties": {"data_size": 20119, "index_size": 84, "filter_size": 72, "raw_key_size": 187, "raw_average_key_size": 23, "raw_value_size": 19896, "raw_average_value_size": 2487, "num_data_blocks": 3, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:52:22.892253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1402] Level-0 flush table #11231: 21189 bytes OK
>2018-07-12 01:52:22.942258 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.908813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11231 started
>2018-07-12 01:52:22.942266 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.942223) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11231: memtable #1 done
>2018-07-12 01:52:22.942268 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.942238) EVENT_LOG_v1 {"time_micros": 1531360342942232, "job": 1402, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:52:22.942270 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.942249) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:52:22.942275 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1402] Try to delete WAL files size 20070, prev total WAL file size 20070, number of live WAL files 2.
>
>2018-07-12 01:52:22.942365 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:52:22.942414 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:22.942408) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 01:52:22.942422 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1403] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:52:22.942428 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1402 Base level 0, inputs: [11231(20KB)], [11229(14MB)]
>
>2018-07-12 01:52:22.942440 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360342942431, "job": 1403, "event": "compaction_started", "files_L0": [11231], "files_L1": [11229], "score": -1, "input_data_size": 15417485}
>2018-07-12 01:52:23.126358 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1403] Generated table #11232: 3086 keys, 15416368 bytes
>2018-07-12 01:52:23.126388 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360343126373, "cf_name": "default", "job": 1403, "event": "table_file_creation", "file_number": 11232, "file_size": 15416368, "table_properties": {"data_size": 15298670, "index_size": 67266, "filter_size": 49505, "raw_key_size": 63140, "raw_average_key_size": 20, "raw_value_size": 15207310, "raw_average_value_size": 4927, "num_data_blocks": 1962, "num_entries": 3086, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:52:23.167865 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1403] Compacted 1@0 + 1@1 files to L1 => 15416368 bytes
>2018-07-12 01:52:23.201406 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:23.201339) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 68.4 rd, 68.4 wr, level 1, files in(1, 1) out(1) MB in(0.0, 14.7) out(14.7), read-write-amplify(1455.2) write-amplify(727.6) OK, records in: 3091, records dropped: 5
>
>2018-07-12 01:52:23.201415 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:52:23.201361) EVENT_LOG_v1 {"time_micros": 1531360343201351, "job": 1403, "event": "compaction_finished", "compaction_time_micros": 225410, "output_level": 1, "num_output_files": 1, "total_output_size": 15416368, "num_input_records": 3091, "num_output_records": 3086, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:52:23.201492 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360343201488, "job": 1403, "event": "table_file_deletion", "file_number": 11231}
>2018-07-12 01:52:23.204465 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360343204462, "job": 1403, "event": "table_file_deletion", "file_number": 11229}
>2018-07-12 01:53:12.983692 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:53:27.303388 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:53:27.458053 7f29149e4700  1 mon.magna102@0(leader).log v22349 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22349)
>2018-07-12 01:53:35.306822 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:53:35.458660 7f29149e4700  1 mon.magna102@0(leader).log v22350 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22350)
>2018-07-12 01:53:53.314697 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 10 (REQUEST_SLOW)
>2018-07-12 01:53:53.486367 7f29149e4700  1 mon.magna102@0(leader).log v22351 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22351)
>2018-07-12 01:54:02.368292 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:54:02.478781 7f29149e4700  1 mon.magna102@0(leader).log v22352 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22352)
>2018-07-12 01:54:12.983871 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:55:03.346165 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:55:03.501073 7f29149e4700  1 mon.magna102@0(leader).log v22353 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22353)
>2018-07-12 01:55:11.349763 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:55:11.501702 7f29149e4700  1 mon.magna102@0(leader).log v22354 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22354)
>2018-07-12 01:55:12.984055 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:55:21.494518 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11233. Immutable memtables: 0.
>
>2018-07-12 01:55:21.494562 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1404] Syncing log #11230
>2018-07-12 01:55:21.511059 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.494552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 01:55:21.511070 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1404] Flushing memtable with next log file: 11233
>
>2018-07-12 01:55:21.511097 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521511089, "job": 1404, "event": "flush_started", "num_memtables": 1, "num_entries": 934, "num_deletes": 251, "memory_usage": 1521408}
>2018-07-12 01:55:21.511101 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1404] Level-0 flush table #11234: started
>2018-07-12 01:55:21.561579 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521561565, "cf_name": "default", "job": 1404, "event": "table_file_creation", "file_number": 11234, "file_size": 1442566, "table_properties": {"data_size": 1430095, "index_size": 6365, "filter_size": 5186, "raw_key_size": 9393, "raw_average_key_size": 20, "raw_value_size": 1419193, "raw_average_value_size": 3098, "num_data_blocks": 194, "num_entries": 458, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 01:55:21.561589 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1404] Level-0 flush table #11234: 1442566 bytes OK
>2018-07-12 01:55:21.611632 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.578125) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11234 started
>2018-07-12 01:55:21.611640 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.611596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11234: memtable #1 done
>2018-07-12 01:55:21.611643 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.611610) EVENT_LOG_v1 {"time_micros": 1531360521611604, "job": 1404, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 01:55:21.611645 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.611622) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 01:55:21.611663 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1404] Try to delete WAL files size 1501979, prev total WAL file size 1501979, number of live WAL files 2.
>
>2018-07-12 01:55:21.612112 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 01:55:21.612174 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.612167) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 01:55:21.612194 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1405] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 01:55:21.612201 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1404 Base level 0, inputs: [11234(1408KB)], [11232(14MB)]
>
>2018-07-12 01:55:21.612214 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521612204, "job": 1405, "event": "compaction_started", "files_L0": [11234], "files_L1": [11232], "score": -1, "input_data_size": 16858934}
>2018-07-12 01:55:21.795652 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1405] Generated table #11235: 3034 keys, 14909001 bytes
>2018-07-12 01:55:21.795695 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521795669, "cf_name": "default", "job": 1405, "event": "table_file_creation", "file_number": 11235, "file_size": 14909001, "table_properties": {"data_size": 14794503, "index_size": 65557, "filter_size": 48014, "raw_key_size": 62286, "raw_average_key_size": 20, "raw_value_size": 14705009, "raw_average_value_size": 4846, "num_data_blocks": 1904, "num_entries": 3034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 01:55:21.828871 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1405] Compacted 1@0 + 1@1 files to L1 => 14909001 bytes
>2018-07-12 01:55:21.862382 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.862314) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 77.8 rd, 68.8 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.7) out(14.2), read-write-amplify(22.0) write-amplify(10.3) OK, records in: 3544, records dropped: 510
>
>2018-07-12 01:55:21.862391 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-01:55:21.862335) EVENT_LOG_v1 {"time_micros": 1531360521862325, "job": 1405, "event": "compaction_finished", "compaction_time_micros": 216641, "output_level": 1, "num_output_files": 1, "total_output_size": 14909001, "num_input_records": 3544, "num_output_records": 3034, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 01:55:21.862766 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521862762, "job": 1405, "event": "table_file_deletion", "file_number": 11234}
>2018-07-12 01:55:21.865672 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360521865669, "job": 1405, "event": "table_file_deletion", "file_number": 11232}
>2018-07-12 01:55:25.356170 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:55:25.513386 7f29149e4700  1 mon.magna102@0(leader).log v22355 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22355)
>2018-07-12 01:55:32.396901 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:55:32.497272 7f29149e4700  1 mon.magna102@0(leader).log v22356 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22356)
>2018-07-12 01:56:12.984237 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:56:43.391219 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:56:43.537107 7f29149e4700  1 mon.magna102@0(leader).log v22357 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22357)
>2018-07-12 01:56:51.394954 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:56:51.546149 7f29149e4700  1 mon.magna102@0(leader).log v22358 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22358)
>2018-07-12 01:57:01.399412 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check failed: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:57:01.548250 7f29149e4700  1 mon.magna102@0(leader).log v22359 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22359)
>2018-07-12 01:57:07.440170 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 7,10 (REQUEST_SLOW)
>2018-07-12 01:57:07.557208 7f29149e4700  1 mon.magna102@0(leader).log v22360 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22360)
>2018-07-12 01:57:12.984407 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:58:07.428969 7f291b9f2700  0 log_channel(cluster) log [WRN] : Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 7 (REQUEST_SLOW)
>2018-07-12 01:58:07.579393 7f29149e4700  1 mon.magna102@0(leader).log v22361 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22361)
>2018-07-12 01:58:12.984576 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 01:58:16.746716 7f29149e4700  1 mon.magna102@0(leader).log v22362 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22362)
>2018-07-12 01:58:17.880402 7f29149e4700  1 mon.magna102@0(leader).log v22363 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22363)
>2018-07-12 01:58:21.435271 7f291b9f2700  0 log_channel(cluster) log [INF] : Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 7)
>2018-07-12 01:58:21.589314 7f29149e4700  1 mon.magna102@0(leader).log v22364 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 1 entries (version 22364)
>2018-07-12 01:58:22.864442 7f29149e4700  1 mon.magna102@0(leader).log v22365 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22365)
>2018-07-12 01:58:32.865317 7f29149e4700  1 mon.magna102@0(leader).log v22366 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22366)
>2018-07-12 01:58:35.165615 7f29149e4700  1 mon.magna102@0(leader).log v22367 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22367)
>2018-07-12 01:59:12.984746 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:00:12.984913 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:00:42.602904 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11236. Immutable memtables: 0.
>
>2018-07-12 02:00:42.602948 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1406] Syncing log #11233
>2018-07-12 02:00:42.619402 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.602938) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:00:42.619414 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1406] Flushing memtable with next log file: 11236
>
>2018-07-12 02:00:42.619440 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842619432, "job": 1406, "event": "flush_started", "num_memtables": 1, "num_entries": 1479, "num_deletes": 250, "memory_usage": 2706168}
>2018-07-12 02:00:42.619443 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1406] Level-0 flush table #11237: started
>2018-07-12 02:00:42.669942 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842669927, "cf_name": "default", "job": 1406, "event": "table_file_creation", "file_number": 11237, "file_size": 1527518, "table_properties": {"data_size": 1515014, "index_size": 6218, "filter_size": 5366, "raw_key_size": 9854, "raw_average_key_size": 21, "raw_value_size": 1504158, "raw_average_value_size": 3241, "num_data_blocks": 194, "num_entries": 464, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:00:42.669952 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1406] Level-0 flush table #11237: 1527518 bytes OK
>2018-07-12 02:00:42.719968 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.686475) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11237 started
>2018-07-12 02:00:42.719976 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.719932) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11237: memtable #1 done
>2018-07-12 02:00:42.719979 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.719946) EVENT_LOG_v1 {"time_micros": 1531360842719940, "job": 1406, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:00:42.719994 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.719958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:00:42.720001 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1406] Try to delete WAL files size 2676779, prev total WAL file size 2676779, number of live WAL files 2.
>
>2018-07-12 02:00:42.720707 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:00:42.720902 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 02:00:42.720909 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 122869.6 total, 783.3 interval
>Cumulative writes: 150K writes, 700K keys, 150K commit groups, 1.0 writes per commit group, ingest: 1.14 GB, 0.01 MB/s
>Cumulative WAL: 150K writes, 150K syncs, 1.00 writes per sync, written: 1.14 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 850 writes, 3751 keys, 850 commit groups, 1.0 writes per commit group, ingest: 6.27 MB, 0.01 MB/s
>Interval WAL: 850 writes, 850 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.46 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       704    0.077       0      0
>  L1      1/1   14.22 MB   0.0     11.1     0.9     10.1      10.1     -0.0       0.0  10.8     72.5     66.4       157       701    0.223   2518K   307K
> Sum      2/2   15.68 MB   0.0     11.1     0.9     10.1      11.1      0.9       0.0  11.8     53.9     53.9       211      1405    0.150   2518K   307K
> Int      0/0    0.00 KB   0.0      0.1     0.0      0.1       0.1      0.0       0.0  14.9     52.9     52.6         1         8    0.147     13K   1398
>Uptime(secs): 122869.6 total, 122869.6 interval
>Flush(GB): cumulative 0.943, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.09 GB write, 0.09 MB/s write, 11.09 GB read, 0.09 MB/s read, 210.8 seconds
>Interval compaction: 0.06 GB write, 0.00 MB/s write, 0.06 GB read, 0.00 MB/s read, 1.2 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.46 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       704    0.077       0      0
>  L1      1/1   14.22 MB   0.0     11.1     0.9     10.1      10.1     -0.0       0.0  10.8     72.5     66.4       157       701    0.223   2518K   307K
> Sum      2/2   15.68 MB   0.0     11.1     0.9     10.1      11.1      0.9       0.0  11.8     53.9     53.9       211      1405    0.150   2518K   307K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 122869.6 total, 122869.6 interval
>Flush(GB): cumulative 0.943, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.09 GB write, 0.09 MB/s write, 11.09 GB read, 0.09 MB/s read, 210.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 02:00:42.720930 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.720924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:00:42.720935 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1407] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:00:42.720941 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1406 Base level 0, inputs: [11237(1491KB)], [11235(14MB)]
>
>2018-07-12 02:00:42.720953 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842720944, "job": 1407, "event": "compaction_started", "files_L0": [11237], "files_L1": [11235], "score": -1, "input_data_size": 16436519}
>2018-07-12 02:00:42.912333 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1407] Generated table #11238: 3144 keys, 15773219 bytes
>2018-07-12 02:00:42.912384 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842912367, "cf_name": "default", "job": 1407, "event": "table_file_creation", "file_number": 11238, "file_size": 15773219, "table_properties": {"data_size": 15653057, "index_size": 68638, "filter_size": 50597, "raw_key_size": 64319, "raw_average_key_size": 20, "raw_value_size": 15559912, "raw_average_value_size": 4949, "num_data_blocks": 2002, "num_entries": 3144, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:00:42.945533 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1407] Compacted 1@0 + 1@1 files to L1 => 15773219 bytes
>2018-07-12 02:00:42.979067 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.978994) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 73.2 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.2) out(15.0), read-write-amplify(21.1) write-amplify(10.3) OK, records in: 3498, records dropped: 354
>
>2018-07-12 02:00:42.979075 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:00:42.979015) EVENT_LOG_v1 {"time_micros": 1531360842979005, "job": 1407, "event": "compaction_finished", "compaction_time_micros": 224562, "output_level": 1, "num_output_files": 1, "total_output_size": 15773219, "num_input_records": 3498, "num_output_records": 3144, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:00:42.979436 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842979430, "job": 1407, "event": "table_file_deletion", "file_number": 11237}
>2018-07-12 02:00:42.982341 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360842982325, "job": 1407, "event": "table_file_deletion", "file_number": 11235}
>2018-07-12 02:00:42.982372 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:01:12.985090 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:02:12.985265 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:03:12.985452 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:03:13.709727 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11239. Immutable memtables: 0.
>
>2018-07-12 02:03:13.709774 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1408] Syncing log #11236
>2018-07-12 02:03:13.726293 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.709764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:03:13.726304 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1408] Flushing memtable with next log file: 11239
>
>2018-07-12 02:03:13.726331 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360993726323, "job": 1408, "event": "flush_started", "num_memtables": 1, "num_entries": 788, "num_deletes": 251, "memory_usage": 1152000}
>2018-07-12 02:03:13.726334 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1408] Level-0 flush table #11240: started
>2018-07-12 02:03:13.768516 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360993768502, "cf_name": "default", "job": 1408, "event": "table_file_creation", "file_number": 11240, "file_size": 1095989, "table_properties": {"data_size": 1085881, "index_size": 5046, "filter_size": 4142, "raw_key_size": 8388, "raw_average_key_size": 20, "raw_value_size": 1076642, "raw_average_value_size": 2625, "num_data_blocks": 154, "num_entries": 410, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:03:13.768527 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1408] Level-0 flush table #11240: 1095989 bytes OK
>2018-07-12 02:03:13.818526 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.785051) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11240 started
>2018-07-12 02:03:13.818534 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.818490) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11240: memtable #1 done
>2018-07-12 02:03:13.818537 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.818505) EVENT_LOG_v1 {"time_micros": 1531360993818499, "job": 1408, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:03:13.818552 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.818516) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:03:13.818558 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1408] Try to delete WAL files size 1134765, prev total WAL file size 1134765, number of live WAL files 2.
>
>2018-07-12 02:03:13.818976 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:03:13.819039 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:03:13.819031) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:03:13.819046 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1409] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:03:13.819052 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1408 Base level 0, inputs: [11240(1070KB)], [11238(15MB)]
>
>2018-07-12 02:03:13.819066 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360993819055, "job": 1409, "event": "compaction_started", "files_L0": [11240], "files_L1": [11238], "score": -1, "input_data_size": 16869208}
>2018-07-12 02:03:14.002654 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1409] Generated table #11241: 3046 keys, 14892393 bytes
>2018-07-12 02:03:14.002702 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360994002674, "cf_name": "default", "job": 1409, "event": "table_file_creation", "file_number": 11241, "file_size": 14892393, "table_properties": {"data_size": 14777892, "index_size": 65567, "filter_size": 48007, "raw_key_size": 62511, "raw_average_key_size": 20, "raw_value_size": 14688221, "raw_average_value_size": 4822, "num_data_blocks": 1904, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:03:14.044100 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1409] Compacted 1@0 + 1@1 files to L1 => 14892393 bytes
>2018-07-12 02:03:14.077634 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:03:14.077567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.0 rd, 66.2 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.0) out(14.2), read-write-amplify(29.0) write-amplify(13.6) OK, records in: 3554, records dropped: 508
>
>2018-07-12 02:03:14.077642 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:03:14.077588) EVENT_LOG_v1 {"time_micros": 1531360994077579, "job": 1409, "event": "compaction_finished", "compaction_time_micros": 225020, "output_level": 1, "num_output_files": 1, "total_output_size": 14892393, "num_input_records": 3554, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:03:14.077945 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360994077940, "job": 1409, "event": "table_file_deletion", "file_number": 11240}
>2018-07-12 02:03:14.080865 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531360994080862, "job": 1409, "event": "table_file_deletion", "file_number": 11238}
>2018-07-12 02:04:12.985627 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:05:12.985805 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:06:12.985987 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:07:12.986152 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:07:43.301478 7f29149e4700  1 mon.magna102@0(leader).log v22368 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22368)
>2018-07-12 02:07:53.302322 7f29149e4700  1 mon.magna102@0(leader).log v22369 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22369)
>2018-07-12 02:08:12.986337 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:09:02.817577 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11242. Immutable memtables: 0.
>
>2018-07-12 02:09:02.817620 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1410] Syncing log #11239
>2018-07-12 02:09:02.834075 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.817611) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:09:02.834086 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1410] Flushing memtable with next log file: 11242
>
>2018-07-12 02:09:02.834100 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361342834092, "job": 1410, "event": "flush_started", "num_memtables": 1, "num_entries": 1489, "num_deletes": 250, "memory_usage": 2649960}
>2018-07-12 02:09:02.834102 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1410] Level-0 flush table #11243: started
>2018-07-12 02:09:02.884612 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361342884598, "cf_name": "default", "job": 1410, "event": "table_file_creation", "file_number": 11243, "file_size": 1389395, "table_properties": {"data_size": 1377547, "index_size": 5930, "filter_size": 4998, "raw_key_size": 9463, "raw_average_key_size": 21, "raw_value_size": 1367162, "raw_average_value_size": 3072, "num_data_blocks": 186, "num_entries": 445, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:09:02.884623 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1410] Level-0 flush table #11243: 1389395 bytes OK
>2018-07-12 02:09:02.934668 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.901189) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11243 started
>2018-07-12 02:09:02.934676 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.934632) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11243: memtable #1 done
>2018-07-12 02:09:02.934678 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.934646) EVENT_LOG_v1 {"time_micros": 1531361342934640, "job": 1410, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:09:02.934680 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.934658) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:09:02.934687 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1410] Try to delete WAL files size 2619735, prev total WAL file size 2619735, number of live WAL files 2.
>
>2018-07-12 02:09:02.935341 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:09:02.935394 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:09:02.935387) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:09:02.935403 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1411] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:09:02.935409 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1410 Base level 0, inputs: [11243(1356KB)], [11241(14MB)]
>
>2018-07-12 02:09:02.935421 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361342935412, "job": 1411, "event": "compaction_started", "files_L0": [11243], "files_L1": [11241], "score": -1, "input_data_size": 16281788}
>2018-07-12 02:09:03.118710 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1411] Generated table #11244: 3151 keys, 15720515 bytes
>2018-07-12 02:09:03.118751 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361343118725, "cf_name": "default", "job": 1411, "event": "table_file_creation", "file_number": 11244, "file_size": 15720515, "table_properties": {"data_size": 15600232, "index_size": 68809, "filter_size": 50547, "raw_key_size": 64461, "raw_average_key_size": 20, "raw_value_size": 15506872, "raw_average_value_size": 4921, "num_data_blocks": 2007, "num_entries": 3151, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:09:03.151903 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1411] Compacted 1@0 + 1@1 files to L1 => 15720515 bytes
>2018-07-12 02:09:03.185352 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:09:03.185283) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.2 rd, 72.6 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.2) out(15.0), read-write-amplify(23.0) write-amplify(11.3) OK, records in: 3491, records dropped: 340
>
>2018-07-12 02:09:03.185362 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:09:03.185306) EVENT_LOG_v1 {"time_micros": 1531361343185296, "job": 1411, "event": "compaction_finished", "compaction_time_micros": 216467, "output_level": 1, "num_output_files": 1, "total_output_size": 15720515, "num_input_records": 3491, "num_output_records": 3151, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:09:03.185708 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361343185704, "job": 1411, "event": "table_file_deletion", "file_number": 11243}
>2018-07-12 02:09:03.188593 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361343188590, "job": 1411, "event": "table_file_deletion", "file_number": 11241}
>2018-07-12 02:09:03.188644 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:09:12.986520 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:10:12.986692 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:11:12.922608 7f29149e4700  1 mon.magna102@0(leader).log v22370 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22370)
>2018-07-12 02:11:12.986875 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:11:25.915657 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11245. Immutable memtables: 0.
>
>2018-07-12 02:11:25.915702 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1412] Syncing log #11242
>2018-07-12 02:11:25.932231 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:11:25.915693) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:11:25.932242 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1412] Flushing memtable with next log file: 11245
>
>2018-07-12 02:11:25.932269 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361485932260, "job": 1412, "event": "flush_started", "num_memtables": 1, "num_entries": 768, "num_deletes": 251, "memory_usage": 1110800}
>2018-07-12 02:11:25.932272 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1412] Level-0 flush table #11246: started
>2018-07-12 02:11:25.982708 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361485982694, "cf_name": "default", "job": 1412, "event": "table_file_creation", "file_number": 11246, "file_size": 1057393, "table_properties": {"data_size": 1047614, "index_size": 4847, "filter_size": 4012, "raw_key_size": 8332, "raw_average_key_size": 20, "raw_value_size": 1038530, "raw_average_value_size": 2551, "num_data_blocks": 148, "num_entries": 407, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:11:25.982718 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1412] Level-0 flush table #11246: 1057393 bytes OK
>2018-07-12 02:11:26.041096 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:11:25.999293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11246 started
>2018-07-12 02:11:26.041104 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.041060) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11246: memtable #1 done
>2018-07-12 02:11:26.041106 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.041074) EVENT_LOG_v1 {"time_micros": 1531361486041068, "job": 1412, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:11:26.041108 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.041086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:11:26.041127 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1412] Try to delete WAL files size 1093955, prev total WAL file size 1093955, number of live WAL files 2.
>
>2018-07-12 02:11:26.041448 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:11:26.041630 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 02:11:26.041636 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 123512.9 total, 643.3 interval
>Cumulative writes: 150K writes, 703K keys, 150K commit groups, 1.0 writes per commit group, ingest: 1.14 GB, 0.01 MB/s
>Cumulative WAL: 150K writes, 150K syncs, 1.00 writes per sync, written: 1.14 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 656 writes, 3045 keys, 656 commit groups, 1.0 writes per commit group, ingest: 4.62 MB, 0.01 MB/s
>Interval WAL: 656 writes, 656 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.01 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       707    0.077       0      0
>  L1      1/1   14.99 MB   0.0     11.1     0.9     10.2      10.2     -0.0       0.0  10.8     72.5     66.4       157       704    0.223   2529K   308K
> Sum      2/2   16.00 MB   0.0     11.1     0.9     10.2      11.1      0.9       0.0  11.8     53.9     53.9       212      1411    0.150   2529K   308K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  14.1     55.1     55.4         1         6    0.143     10K   1202
>Uptime(secs): 123512.9 total, 123512.9 interval
>Flush(GB): cumulative 0.946, interval 0.003
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.13 GB write, 0.09 MB/s write, 11.13 GB read, 0.09 MB/s read, 211.7 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.01 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        54       707    0.077       0      0
>  L1      1/1   14.99 MB   0.0     11.1     0.9     10.2      10.2     -0.0       0.0  10.8     72.5     66.4       157       704    0.223   2529K   308K
> Sum      2/2   16.00 MB   0.0     11.1     0.9     10.2      11.1      0.9       0.0  11.8     53.9     53.9       212      1411    0.150   2529K   308K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 123512.9 total, 123512.9 interval
>Flush(GB): cumulative 0.946, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.13 GB write, 0.09 MB/s write, 11.13 GB read, 0.09 MB/s read, 211.7 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 02:11:26.041660 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.041654) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:11:26.041666 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1413] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:11:26.041671 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1412 Base level 0, inputs: [11246(1032KB)], [11244(14MB)]
>
>2018-07-12 02:11:26.041683 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361486041674, "job": 1413, "event": "compaction_started", "files_L0": [11246], "files_L1": [11244], "score": -1, "input_data_size": 16777908}
>2018-07-12 02:11:26.218085 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1413] Generated table #11247: 3048 keys, 14798251 bytes
>2018-07-12 02:11:26.218133 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361486218106, "cf_name": "default", "job": 1413, "event": "table_file_creation", "file_number": 11247, "file_size": 14798251, "table_properties": {"data_size": 14683968, "index_size": 65535, "filter_size": 47821, "raw_key_size": 62546, "raw_average_key_size": 20, "raw_value_size": 14594295, "raw_average_value_size": 4788, "num_data_blocks": 1903, "num_entries": 3048, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:11:26.251249 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1413] Compacted 1@0 + 1@1 files to L1 => 14798251 bytes
>2018-07-12 02:11:26.284805 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.284733) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 80.1 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(1.0, 15.0) out(14.1), read-write-amplify(29.9) write-amplify(14.0) OK, records in: 3558, records dropped: 510
>
>2018-07-12 02:11:26.284814 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:11:26.284754) EVENT_LOG_v1 {"time_micros": 1531361486284745, "job": 1413, "event": "compaction_finished", "compaction_time_micros": 209549, "output_level": 1, "num_output_files": 1, "total_output_size": 14798251, "num_input_records": 3558, "num_output_records": 3048, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:11:26.285111 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361486285107, "job": 1413, "event": "table_file_deletion", "file_number": 11246}
>2018-07-12 02:11:26.288030 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361486288027, "job": 1413, "event": "table_file_deletion", "file_number": 11244}
>2018-07-12 02:12:12.987062 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:13:12.987231 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:14:12.987417 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:15:12.987595 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:16:12.987772 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:17:12.987959 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:17:23.031533 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11248. Immutable memtables: 0.
>
>2018-07-12 02:17:23.031576 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1414] Syncing log #11245
>2018-07-12 02:17:23.047996 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.031566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:17:23.048007 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1414] Flushing memtable with next log file: 11248
>
>2018-07-12 02:17:23.048034 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843048026, "job": 1414, "event": "flush_started", "num_memtables": 1, "num_entries": 1501, "num_deletes": 250, "memory_usage": 2672248}
>2018-07-12 02:17:23.048037 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1414] Level-0 flush table #11249: started
>2018-07-12 02:17:23.098537 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843098523, "cf_name": "default", "job": 1414, "event": "table_file_creation", "file_number": 11249, "file_size": 1382645, "table_properties": {"data_size": 1370777, "index_size": 5960, "filter_size": 4988, "raw_key_size": 9370, "raw_average_key_size": 21, "raw_value_size": 1360450, "raw_average_value_size": 3084, "num_data_blocks": 187, "num_entries": 441, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:17:23.098547 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1414] Level-0 flush table #11249: 1382645 bytes OK
>2018-07-12 02:17:23.149800 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.115042) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11249 started
>2018-07-12 02:17:23.149808 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.149763) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11249: memtable #1 done
>2018-07-12 02:17:23.149810 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.149778) EVENT_LOG_v1 {"time_micros": 1531361843149772, "job": 1414, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:17:23.149825 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.149789) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:17:23.149832 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1414] Try to delete WAL files size 2642213, prev total WAL file size 2642213, number of live WAL files 2.
>
>2018-07-12 02:17:23.150534 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:17:23.150598 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.150591) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:17:23.150608 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1415] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:17:23.150614 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1414 Base level 0, inputs: [11249(1350KB)], [11247(14MB)]
>
>2018-07-12 02:17:23.150628 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843150618, "job": 1415, "event": "compaction_started", "files_L0": [11249], "files_L1": [11247], "score": -1, "input_data_size": 16180896}
>2018-07-12 02:17:23.333834 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1415] Generated table #11250: 3155 keys, 15646719 bytes
>2018-07-12 02:17:23.333878 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843333851, "cf_name": "default", "job": 1415, "event": "table_file_creation", "file_number": 11250, "file_size": 15646719, "table_properties": {"data_size": 15526450, "index_size": 68910, "filter_size": 50432, "raw_key_size": 64542, "raw_average_key_size": 20, "raw_value_size": 15432962, "raw_average_value_size": 4891, "num_data_blocks": 2010, "num_entries": 3155, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:17:23.367033 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1415] Compacted 1@0 + 1@1 files to L1 => 15646719 bytes
>2018-07-12 02:17:23.408890 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.408824) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.8 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.1) out(14.9), read-write-amplify(23.0) write-amplify(11.3) OK, records in: 3489, records dropped: 334
>
>2018-07-12 02:17:23.408899 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:17:23.408845) EVENT_LOG_v1 {"time_micros": 1531361843408835, "job": 1415, "event": "compaction_finished", "compaction_time_micros": 216388, "output_level": 1, "num_output_files": 1, "total_output_size": 15646719, "num_input_records": 3489, "num_output_records": 3155, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:17:23.409268 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843409264, "job": 1415, "event": "table_file_deletion", "file_number": 11249}
>2018-07-12 02:17:23.412129 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361843412126, "job": 1415, "event": "table_file_deletion", "file_number": 11247}
>2018-07-12 02:17:23.412248 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:18:12.988148 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:19:12.988334 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:19:44.130696 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11251. Immutable memtables: 0.
>
>2018-07-12 02:19:44.130741 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1416] Syncing log #11248
>2018-07-12 02:19:44.147278 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.130731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:19:44.147290 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1416] Flushing memtable with next log file: 11251
>
>2018-07-12 02:19:44.147317 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984147309, "job": 1416, "event": "flush_started", "num_memtables": 1, "num_entries": 753, "num_deletes": 251, "memory_usage": 1077584}
>2018-07-12 02:19:44.147320 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1416] Level-0 flush table #11252: started
>2018-07-12 02:19:44.197750 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984197737, "cf_name": "default", "job": 1416, "event": "table_file_creation", "file_number": 11252, "file_size": 1024827, "table_properties": {"data_size": 1015277, "index_size": 4718, "filter_size": 3912, "raw_key_size": 8178, "raw_average_key_size": 20, "raw_value_size": 1006417, "raw_average_value_size": 2516, "num_data_blocks": 144, "num_entries": 400, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:19:44.197761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1416] Level-0 flush table #11252: 1024827 bytes OK
>2018-07-12 02:19:44.247847 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.214319) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11252 started
>2018-07-12 02:19:44.247855 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.247811) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11252: memtable #1 done
>2018-07-12 02:19:44.247858 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.247825) EVENT_LOG_v1 {"time_micros": 1531361984247819, "job": 1416, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:19:44.247873 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.247836) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:19:44.247880 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1416] Try to delete WAL files size 1060920, prev total WAL file size 1060920, number of live WAL files 2.
>
>2018-07-12 02:19:44.248222 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:19:44.248277 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.248269) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:19:44.248286 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1417] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:19:44.248292 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1416 Base level 0, inputs: [11252(1000KB)], [11250(14MB)]
>
>2018-07-12 02:19:44.248305 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984248295, "job": 1417, "event": "compaction_started", "files_L0": [11252], "files_L1": [11250], "score": -1, "input_data_size": 16671546}
>2018-07-12 02:19:44.423561 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1417] Generated table #11253: 3047 keys, 14737021 bytes
>2018-07-12 02:19:44.423610 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984423583, "cf_name": "default", "job": 1417, "event": "table_file_creation", "file_number": 11253, "file_size": 14737021, "table_properties": {"data_size": 14622901, "index_size": 65501, "filter_size": 47692, "raw_key_size": 62524, "raw_average_key_size": 20, "raw_value_size": 14533267, "raw_average_value_size": 4769, "num_data_blocks": 1902, "num_entries": 3047, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:19:44.456725 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1417] Compacted 1@0 + 1@1 files to L1 => 14737021 bytes
>2018-07-12 02:19:44.490271 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.490201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 80.0 rd, 70.7 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.9) out(14.1), read-write-amplify(30.6) write-amplify(14.4) OK, records in: 3555, records dropped: 508
>
>2018-07-12 02:19:44.490280 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:19:44.490223) EVENT_LOG_v1 {"time_micros": 1531361984490214, "job": 1417, "event": "compaction_finished", "compaction_time_micros": 208403, "output_level": 1, "num_output_files": 1, "total_output_size": 14737021, "num_input_records": 3555, "num_output_records": 3047, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:19:44.490564 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984490560, "job": 1417, "event": "table_file_deletion", "file_number": 11252}
>2018-07-12 02:19:44.493490 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531361984493487, "job": 1417, "event": "table_file_deletion", "file_number": 11250}
>2018-07-12 02:20:12.988525 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:21:12.988707 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:22:12.988883 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:23:12.989061 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:24:12.989239 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:25:12.989407 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:25:43.255705 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11254. Immutable memtables: 0.
>
>2018-07-12 02:25:43.255749 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1418] Syncing log #11251
>2018-07-12 02:25:43.272212 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.255739) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:25:43.272224 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1418] Flushing memtable with next log file: 11254
>
>2018-07-12 02:25:43.272250 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343272242, "job": 1418, "event": "flush_started", "num_memtables": 1, "num_entries": 1515, "num_deletes": 250, "memory_usage": 2689336}
>2018-07-12 02:25:43.272253 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1418] Level-0 flush table #11255: started
>2018-07-12 02:25:43.322758 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343322744, "cf_name": "default", "job": 1418, "event": "table_file_creation", "file_number": 11255, "file_size": 1391958, "table_properties": {"data_size": 1380031, "index_size": 5992, "filter_size": 5015, "raw_key_size": 9480, "raw_average_key_size": 21, "raw_value_size": 1369585, "raw_average_value_size": 3070, "num_data_blocks": 188, "num_entries": 446, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:25:43.322768 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1418] Level-0 flush table #11255: 1391958 bytes OK
>2018-07-12 02:25:43.372774 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.339272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11255 started
>2018-07-12 02:25:43.372782 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.372738) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11255: memtable #1 done
>2018-07-12 02:25:43.372786 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.372753) EVENT_LOG_v1 {"time_micros": 1531362343372746, "job": 1418, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:25:43.372788 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.372765) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:25:43.372807 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1418] Try to delete WAL files size 2658934, prev total WAL file size 2658934, number of live WAL files 2.
>
>2018-07-12 02:25:43.373531 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:25:43.373730 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 02:25:43.373736 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 124370.2 total, 857.3 interval
>Cumulative writes: 151K writes, 707K keys, 151K commit groups, 1.0 writes per commit group, ingest: 1.15 GB, 0.01 MB/s
>Cumulative WAL: 151K writes, 151K syncs, 1.00 writes per sync, written: 1.15 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 864 writes, 3769 keys, 864 commit groups, 1.0 writes per commit group, ingest: 6.07 MB, 0.01 MB/s
>Interval WAL: 864 writes, 864 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.33 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        55       710    0.077       0      0
>  L1      1/1   14.05 MB   0.0     11.2     0.9     10.2      10.2     -0.0       0.0  10.8     72.5     66.4       158       707    0.223   2539K   310K
> Sum      2/2   15.38 MB   0.0     11.2     0.9     10.2      11.2      0.9       0.0  11.8     53.9     53.9       212      1417    0.150   2539K   310K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.9     56.7     55.9         1         6    0.139     10K   1352
>Uptime(secs): 124370.2 total, 124370.2 interval
>Flush(GB): cumulative 0.950, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.18 GB write, 0.09 MB/s write, 11.18 GB read, 0.09 MB/s read, 212.5 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.33 MB   0.0      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     17.8        55       710    0.077       0      0
>  L1      1/1   14.05 MB   0.0     11.2     0.9     10.2      10.2     -0.0       0.0  10.8     72.5     66.4       158       707    0.223   2539K   310K
> Sum      2/2   15.38 MB   0.0     11.2     0.9     10.2      11.2      0.9       0.0  11.8     53.9     53.9       212      1417    0.150   2539K   310K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 124370.2 total, 124370.2 interval
>Flush(GB): cumulative 0.950, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.18 GB write, 0.09 MB/s write, 11.18 GB read, 0.09 MB/s read, 212.5 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 02:25:43.373758 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.373751) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:25:43.373765 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1419] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:25:43.373771 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1418 Base level 0, inputs: [11255(1359KB)], [11253(14MB)]
>
>2018-07-12 02:25:43.373783 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343373774, "job": 1419, "event": "compaction_started", "files_L0": [11255], "files_L1": [11253], "score": -1, "input_data_size": 16128979}
>2018-07-12 02:25:43.565159 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1419] Generated table #11256: 3158 keys, 15601574 bytes
>2018-07-12 02:25:43.565224 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343565208, "cf_name": "default", "job": 1419, "event": "table_file_creation", "file_number": 11256, "file_size": 15601574, "table_properties": {"data_size": 15481320, "index_size": 68970, "filter_size": 50357, "raw_key_size": 64598, "raw_average_key_size": 20, "raw_value_size": 15387751, "raw_average_value_size": 4872, "num_data_blocks": 2012, "num_entries": 3158, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:25:43.598350 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1419] Compacted 1@0 + 1@1 files to L1 => 15601574 bytes
>2018-07-12 02:25:43.631846 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.631775) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.8 rd, 69.5 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.1) out(14.9), read-write-amplify(22.8) write-amplify(11.2) OK, records in: 3493, records dropped: 335
>
>2018-07-12 02:25:43.631854 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:25:43.631796) EVENT_LOG_v1 {"time_micros": 1531362343631787, "job": 1419, "event": "compaction_finished", "compaction_time_micros": 224550, "output_level": 1, "num_output_files": 1, "total_output_size": 15601574, "num_input_records": 3493, "num_output_records": 3158, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:25:43.632226 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343632222, "job": 1419, "event": "table_file_deletion", "file_number": 11255}
>2018-07-12 02:25:43.635070 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362343635067, "job": 1419, "event": "table_file_deletion", "file_number": 11253}
>2018-07-12 02:25:43.635202 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:26:12.989590 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:27:12.989773 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:28:00.369356 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11257. Immutable memtables: 0.
>
>2018-07-12 02:28:00.369400 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1420] Syncing log #11254
>2018-07-12 02:28:00.385906 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.369390) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:28:00.385918 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1420] Flushing memtable with next log file: 11257
>
>2018-07-12 02:28:00.385945 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480385937, "job": 1420, "event": "flush_started", "num_memtables": 1, "num_entries": 739, "num_deletes": 251, "memory_usage": 1047784}
>2018-07-12 02:28:00.385948 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1420] Level-0 flush table #11258: started
>2018-07-12 02:28:00.436434 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480436420, "cf_name": "default", "job": 1420, "event": "table_file_creation", "file_number": 11258, "file_size": 996363, "table_properties": {"data_size": 987036, "index_size": 4587, "filter_size": 3820, "raw_key_size": 8094, "raw_average_key_size": 20, "raw_value_size": 978327, "raw_average_value_size": 2470, "num_data_blocks": 140, "num_entries": 396, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:28:00.436444 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1420] Level-0 flush table #11258: 996363 bytes OK
>2018-07-12 02:28:00.486446 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.452960) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11258 started
>2018-07-12 02:28:00.486454 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.486411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11258: memtable #1 done
>2018-07-12 02:28:00.486457 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.486425) EVENT_LOG_v1 {"time_micros": 1531362480486419, "job": 1420, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:28:00.486459 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.486436) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:28:00.486478 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1420] Try to delete WAL files size 1031382, prev total WAL file size 1031382, number of live WAL files 2.
>
>2018-07-12 02:28:00.486806 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:28:00.486858 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.486851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:28:00.486868 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1421] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:28:00.486874 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1420 Base level 0, inputs: [11258(973KB)], [11256(14MB)]
>
>2018-07-12 02:28:00.486887 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480486877, "job": 1421, "event": "compaction_started", "files_L0": [11258], "files_L1": [11256], "score": -1, "input_data_size": 16597937}
>2018-07-12 02:28:00.670512 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1421] Generated table #11259: 3046 keys, 14712527 bytes
>2018-07-12 02:28:00.670559 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480670532, "cf_name": "default", "job": 1421, "event": "table_file_creation", "file_number": 11259, "file_size": 14712527, "table_properties": {"data_size": 14598551, "index_size": 65424, "filter_size": 47625, "raw_key_size": 62496, "raw_average_key_size": 20, "raw_value_size": 14508988, "raw_average_value_size": 4763, "num_data_blocks": 1900, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:28:00.712003 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1421] Compacted 1@0 + 1@1 files to L1 => 14712527 bytes
>2018-07-12 02:28:00.745523 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.745457) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 73.7 rd, 65.4 wr, level 1, files in(1, 1) out(1) MB in(1.0, 14.9) out(14.0), read-write-amplify(31.4) write-amplify(14.8) OK, records in: 3554, records dropped: 508
>
>2018-07-12 02:28:00.745532 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:28:00.745479) EVENT_LOG_v1 {"time_micros": 1531362480745469, "job": 1421, "event": "compaction_finished", "compaction_time_micros": 225100, "output_level": 1, "num_output_files": 1, "total_output_size": 14712527, "num_input_records": 3554, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:28:00.745815 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480745810, "job": 1421, "event": "table_file_deletion", "file_number": 11258}
>2018-07-12 02:28:00.748738 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362480748735, "job": 1421, "event": "table_file_deletion", "file_number": 11256}
>2018-07-12 02:28:12.989956 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:29:12.990111 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:30:12.990285 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:31:12.990464 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:32:12.990640 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:33:12.990797 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:34:03.460295 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11260. Immutable memtables: 0.
>
>2018-07-12 02:34:03.460339 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1422] Syncing log #11257
>2018-07-12 02:34:03.476721 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.460329) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:34:03.476732 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1422] Flushing memtable with next log file: 11260
>
>2018-07-12 02:34:03.476758 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843476750, "job": 1422, "event": "flush_started", "num_memtables": 1, "num_entries": 1522, "num_deletes": 250, "memory_usage": 2717232}
>2018-07-12 02:34:03.476761 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1422] Level-0 flush table #11261: started
>2018-07-12 02:34:03.535609 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843535595, "cf_name": "default", "job": 1422, "event": "table_file_creation", "file_number": 11261, "file_size": 1405012, "table_properties": {"data_size": 1392977, "index_size": 6056, "filter_size": 5059, "raw_key_size": 9430, "raw_average_key_size": 21, "raw_value_size": 1382539, "raw_average_value_size": 3113, "num_data_blocks": 190, "num_entries": 444, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:34:03.535620 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1422] Level-0 flush table #11261: 1405012 bytes OK
>2018-07-12 02:34:03.585557 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.552142) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11261 started
>2018-07-12 02:34:03.585565 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.585508) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11261: memtable #1 done
>2018-07-12 02:34:03.585568 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.585523) EVENT_LOG_v1 {"time_micros": 1531362843585516, "job": 1422, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:34:03.585570 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.585547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:34:03.585576 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1422] Try to delete WAL files size 2686520, prev total WAL file size 2686520, number of live WAL files 2.
>
>2018-07-12 02:34:03.586276 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:34:03.586325 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.586319) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:34:03.586335 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1423] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:34:03.586341 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1422 Base level 0, inputs: [11261(1372KB)], [11259(14MB)]
>
>2018-07-12 02:34:03.586354 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843586344, "job": 1423, "event": "compaction_started", "files_L0": [11261], "files_L1": [11259], "score": -1, "input_data_size": 16117539}
>2018-07-12 02:34:03.769669 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1423] Generated table #11262: 3159 keys, 15603712 bytes
>2018-07-12 02:34:03.769717 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843769689, "cf_name": "default", "job": 1423, "event": "table_file_creation", "file_number": 11262, "file_size": 15603712, "table_properties": {"data_size": 15483412, "index_size": 69003, "filter_size": 50370, "raw_key_size": 64618, "raw_average_key_size": 20, "raw_value_size": 15389806, "raw_average_value_size": 4871, "num_data_blocks": 2013, "num_entries": 3159, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:34:03.811161 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1423] Compacted 1@0 + 1@1 files to L1 => 15603712 bytes
>2018-07-12 02:34:03.844703 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.844636) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 71.7 rd, 69.4 wr, level 1, files in(1, 1) out(1) MB in(1.3, 14.0) out(14.9), read-write-amplify(22.6) write-amplify(11.1) OK, records in: 3490, records dropped: 331
>
>2018-07-12 02:34:03.844712 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:34:03.844657) EVENT_LOG_v1 {"time_micros": 1531362843844648, "job": 1423, "event": "compaction_finished", "compaction_time_micros": 224792, "output_level": 1, "num_output_files": 1, "total_output_size": 15603712, "num_input_records": 3490, "num_output_records": 3159, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:34:03.845075 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843845071, "job": 1423, "event": "table_file_deletion", "file_number": 11261}
>2018-07-12 02:34:03.847960 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362843847957, "job": 1423, "event": "table_file_deletion", "file_number": 11259}
>2018-07-12 02:34:03.848076 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:34:12.990995 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:35:12.991169 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:35:49.554794 7f29149e4700  1 mon.magna102@0(leader).log v22371 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22371)
>2018-07-12 02:36:12.991355 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:36:16.557048 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11263. Immutable memtables: 0.
>
>2018-07-12 02:36:16.557091 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1424] Syncing log #11260
>2018-07-12 02:36:16.573466 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.557081) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:36:16.573475 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1424] Flushing memtable with next log file: 11263
>
>2018-07-12 02:36:16.573488 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976573481, "job": 1424, "event": "flush_started", "num_memtables": 1, "num_entries": 733, "num_deletes": 251, "memory_usage": 1036136}
>2018-07-12 02:36:16.573491 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1424] Level-0 flush table #11264: started
>2018-07-12 02:36:16.624139 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976624125, "cf_name": "default", "job": 1424, "event": "table_file_creation", "file_number": 11264, "file_size": 985855, "table_properties": {"data_size": 976636, "index_size": 4517, "filter_size": 3782, "raw_key_size": 8122, "raw_average_key_size": 20, "raw_value_size": 967935, "raw_average_value_size": 2438, "num_data_blocks": 138, "num_entries": 397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:36:16.624150 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1424] Level-0 flush table #11264: 985855 bytes OK
>2018-07-12 02:36:16.682487 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.640665) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11264 started
>2018-07-12 02:36:16.682495 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.682451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11264: memtable #1 done
>2018-07-12 02:36:16.682498 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.682466) EVENT_LOG_v1 {"time_micros": 1531362976682459, "job": 1424, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:36:16.682500 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.682478) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:36:16.682519 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1424] Try to delete WAL files size 1019740, prev total WAL file size 1019740, number of live WAL files 2.
>
>2018-07-12 02:36:16.682850 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:36:16.683037 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 02:36:16.683044 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 125003.5 total, 633.3 interval
>Cumulative writes: 152K writes, 710K keys, 152K commit groups, 1.0 writes per commit group, ingest: 1.15 GB, 0.01 MB/s
>Cumulative WAL: 152K writes, 152K syncs, 1.00 writes per sync, written: 1.15 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 642 writes, 2994 keys, 642 commit groups, 1.0 writes per commit group, ingest: 4.52 MB, 0.01 MB/s
>Interval WAL: 642 writes, 642 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   962.75 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       713    0.077       0      0
>  L1      1/1   14.88 MB   0.0     11.2     1.0     10.3      10.3     -0.0       0.0  10.8     72.5     66.4       158       710    0.223   2550K   311K
> Sum      2/2   15.82 MB   0.0     11.2     1.0     10.3      11.2      1.0       0.0  11.8     53.9     53.9       213      1423    0.150   2550K   311K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  14.6     52.7     53.2         1         6    0.147     10K   1174
>Uptime(secs): 125003.5 total, 125003.5 interval
>Flush(GB): cumulative 0.953, interval 0.003
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.23 GB write, 0.09 MB/s write, 11.23 GB read, 0.09 MB/s read, 213.4 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   962.75 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       713    0.077       0      0
>  L1      1/1   14.88 MB   0.0     11.2     1.0     10.3      10.3     -0.0       0.0  10.8     72.5     66.4       158       710    0.223   2550K   311K
> Sum      2/2   15.82 MB   0.0     11.2     1.0     10.3      11.2      1.0       0.0  11.8     53.9     53.9       213      1423    0.150   2550K   311K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 125003.5 total, 125003.5 interval
>Flush(GB): cumulative 0.953, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.23 GB write, 0.09 MB/s write, 11.23 GB read, 0.09 MB/s read, 213.4 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 02:36:16.683065 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.683058) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:36:16.683070 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1425] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:36:16.683075 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1424 Base level 0, inputs: [11264(962KB)], [11262(14MB)]
>
>2018-07-12 02:36:16.683087 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976683078, "job": 1425, "event": "compaction_started", "files_L0": [11264], "files_L1": [11262], "score": -1, "input_data_size": 16589567}
>2018-07-12 02:36:16.858217 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1425] Generated table #11265: 3046 keys, 14709408 bytes
>2018-07-12 02:36:16.858267 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976858250, "cf_name": "default", "job": 1425, "event": "table_file_creation", "file_number": 11265, "file_size": 14709408, "table_properties": {"data_size": 14595481, "index_size": 65389, "filter_size": 47611, "raw_key_size": 62493, "raw_average_key_size": 20, "raw_value_size": 14505943, "raw_average_value_size": 4762, "num_data_blocks": 1899, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:36:16.891386 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1425] Compacted 1@0 + 1@1 files to L1 => 14709408 bytes
>2018-07-12 02:36:16.933254 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.933175) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.6 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.9) out(14.0), read-write-amplify(31.7) write-amplify(14.9) OK, records in: 3556, records dropped: 510
>
>2018-07-12 02:36:16.933263 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:36:16.933203) EVENT_LOG_v1 {"time_micros": 1531362976933194, "job": 1425, "event": "compaction_finished", "compaction_time_micros": 208281, "output_level": 1, "num_output_files": 1, "total_output_size": 14709408, "num_input_records": 3556, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:36:16.933539 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976933535, "job": 1425, "event": "table_file_deletion", "file_number": 11264}
>2018-07-12 02:36:16.936448 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531362976936445, "job": 1425, "event": "table_file_deletion", "file_number": 11262}
>2018-07-12 02:36:17.658804 7f29149e4700  1 mon.magna102@0(leader).log v22372 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22372)
>2018-07-12 02:37:12.991532 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:38:12.991718 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:39:12.991908 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:40:12.992095 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:41:12.992268 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:42:12.992459 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:42:23.698359 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11266. Immutable memtables: 0.
>
>2018-07-12 02:42:23.698403 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1426] Syncing log #11263
>2018-07-12 02:42:23.714784 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.698394) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:42:23.714796 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1426] Flushing memtable with next log file: 11266
>
>2018-07-12 02:42:23.714822 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363343714814, "job": 1426, "event": "flush_started", "num_memtables": 1, "num_entries": 1544, "num_deletes": 250, "memory_usage": 2764328}
>2018-07-12 02:42:23.714825 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1426] Level-0 flush table #11267: started
>2018-07-12 02:42:23.765328 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363343765314, "cf_name": "default", "job": 1426, "event": "table_file_creation", "file_number": 11267, "file_size": 1437467, "table_properties": {"data_size": 1425247, "index_size": 6149, "filter_size": 5151, "raw_key_size": 9582, "raw_average_key_size": 21, "raw_value_size": 1414617, "raw_average_value_size": 3136, "num_data_blocks": 193, "num_entries": 451, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:42:23.765338 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1426] Level-0 flush table #11267: 1437467 bytes OK
>2018-07-12 02:42:23.815359 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.781840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11267 started
>2018-07-12 02:42:23.815367 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.815322) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11267: memtable #1 done
>2018-07-12 02:42:23.815369 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.815337) EVENT_LOG_v1 {"time_micros": 1531363343815330, "job": 1426, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:42:23.815384 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.815349) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:42:23.815391 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1426] Try to delete WAL files size 2733608, prev total WAL file size 2733608, number of live WAL files 2.
>
>2018-07-12 02:42:23.816097 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:42:23.816160 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:42:23.816153) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:42:23.816167 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1427] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:42:23.816173 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1426 Base level 0, inputs: [11267(1403KB)], [11265(14MB)]
>
>2018-07-12 02:42:23.816199 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363343816177, "job": 1427, "event": "compaction_started", "files_L0": [11267], "files_L1": [11265], "score": -1, "input_data_size": 16146875}
>2018-07-12 02:42:23.999399 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1427] Generated table #11268: 3166 keys, 15646608 bytes
>2018-07-12 02:42:23.999448 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363343999432, "cf_name": "default", "job": 1427, "event": "table_file_creation", "file_number": 11268, "file_size": 15646608, "table_properties": {"data_size": 15526010, "index_size": 69171, "filter_size": 50500, "raw_key_size": 64760, "raw_average_key_size": 20, "raw_value_size": 15432187, "raw_average_value_size": 4874, "num_data_blocks": 2018, "num_entries": 3166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:42:24.032574 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1427] Compacted 1@0 + 1@1 files to L1 => 15646608 bytes
>2018-07-12 02:42:24.066079 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:42:24.066011) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 74.6 rd, 72.3 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(14.9), read-write-amplify(22.1) write-amplify(10.9) OK, records in: 3497, records dropped: 331
>
>2018-07-12 02:42:24.066088 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:42:24.066032) EVENT_LOG_v1 {"time_micros": 1531363344066022, "job": 1427, "event": "compaction_finished", "compaction_time_micros": 216361, "output_level": 1, "num_output_files": 1, "total_output_size": 15646608, "num_input_records": 3497, "num_output_records": 3166, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:42:24.066438 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363344066433, "job": 1427, "event": "table_file_deletion", "file_number": 11267}
>2018-07-12 02:42:24.069308 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363344069305, "job": 1427, "event": "table_file_deletion", "file_number": 11265}
>2018-07-12 02:42:24.069352 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:43:12.992642 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:44:12.992835 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:44:32.778002 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11269. Immutable memtables: 0.
>
>2018-07-12 02:44:32.778047 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1428] Syncing log #11266
>2018-07-12 02:44:32.794550 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.778037) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:44:32.794562 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1428] Flushing memtable with next log file: 11269
>
>2018-07-12 02:44:32.794588 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363472794580, "job": 1428, "event": "flush_started", "num_memtables": 1, "num_entries": 711, "num_deletes": 251, "memory_usage": 988064}
>2018-07-12 02:44:32.794592 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1428] Level-0 flush table #11270: started
>2018-07-12 02:44:32.845094 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363472845080, "cf_name": "default", "job": 1428, "event": "table_file_creation", "file_number": 11270, "file_size": 939439, "table_properties": {"data_size": 930553, "index_size": 4326, "filter_size": 3640, "raw_key_size": 7926, "raw_average_key_size": 20, "raw_value_size": 922147, "raw_average_value_size": 2376, "num_data_blocks": 132, "num_entries": 388, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:44:32.845104 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1428] Level-0 flush table #11270: 939439 bytes OK
>2018-07-12 02:44:32.903451 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.861618) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11270 started
>2018-07-12 02:44:32.903459 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.903414) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11270: memtable #1 done
>2018-07-12 02:44:32.903461 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.903429) EVENT_LOG_v1 {"time_micros": 1531363472903422, "job": 1428, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:44:32.903475 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.903440) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:44:32.903482 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1428] Try to delete WAL files size 972306, prev total WAL file size 972306, number of live WAL files 2.
>
>2018-07-12 02:44:32.903805 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:44:32.903856 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:44:32.903849) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:44:32.903863 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1429] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:44:32.903869 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1428 Base level 0, inputs: [11270(917KB)], [11268(14MB)]
>
>2018-07-12 02:44:32.903883 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363472903873, "job": 1429, "event": "compaction_started", "files_L0": [11270], "files_L1": [11268], "score": -1, "input_data_size": 16586047}
>2018-07-12 02:44:33.079159 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1429] Generated table #11271: 3046 keys, 14712434 bytes
>2018-07-12 02:44:33.079220 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363473079204, "cf_name": "default", "job": 1429, "event": "table_file_creation", "file_number": 11271, "file_size": 14712434, "table_properties": {"data_size": 14598497, "index_size": 65389, "filter_size": 47621, "raw_key_size": 62490, "raw_average_key_size": 20, "raw_value_size": 14508960, "raw_average_value_size": 4763, "num_data_blocks": 1899, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:44:33.112297 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1429] Compacted 1@0 + 1@1 files to L1 => 14712434 bytes
>2018-07-12 02:44:33.154144 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:44:33.154077) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.6 rd, 70.6 wr, level 1, files in(1, 1) out(1) MB in(0.9, 14.9) out(14.0), read-write-amplify(33.3) write-amplify(15.7) OK, records in: 3554, records dropped: 508
>
>2018-07-12 02:44:33.154153 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:44:33.154099) EVENT_LOG_v1 {"time_micros": 1531363473154089, "job": 1429, "event": "compaction_finished", "compaction_time_micros": 208396, "output_level": 1, "num_output_files": 1, "total_output_size": 14712434, "num_input_records": 3554, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:44:33.154427 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363473154423, "job": 1429, "event": "table_file_deletion", "file_number": 11270}
>2018-07-12 02:44:33.157410 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363473157395, "job": 1429, "event": "table_file_deletion", "file_number": 11268}
>2018-07-12 02:45:12.993016 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:46:12.993215 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:47:12.993400 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:48:12.993584 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:49:12.993761 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:50:12.993943 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:50:16.608541 7f29149e4700  1 mon.magna102@0(leader).log v22373 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22373)
>2018-07-12 02:50:43.894439 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11272. Immutable memtables: 0.
>
>2018-07-12 02:50:43.894483 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1430] Syncing log #11269
>2018-07-12 02:50:43.910919 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:50:43.894473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:50:43.910931 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1430] Flushing memtable with next log file: 11272
>
>2018-07-12 02:50:43.910957 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363843910948, "job": 1430, "event": "flush_started", "num_memtables": 1, "num_entries": 1558, "num_deletes": 250, "memory_usage": 2794368}
>2018-07-12 02:50:43.910960 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1430] Level-0 flush table #11273: started
>2018-07-12 02:50:43.961441 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363843961428, "cf_name": "default", "job": 1430, "event": "table_file_creation", "file_number": 11273, "file_size": 1452057, "table_properties": {"data_size": 1439727, "index_size": 6213, "filter_size": 5197, "raw_key_size": 9622, "raw_average_key_size": 21, "raw_value_size": 1429025, "raw_average_value_size": 3154, "num_data_blocks": 195, "num_entries": 453, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:50:43.961452 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1430] Level-0 flush table #11273: 1452057 bytes OK
>2018-07-12 02:50:44.019795 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:50:43.977965) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11273 started
>2018-07-12 02:50:44.019803 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.019758) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11273: memtable #1 done
>2018-07-12 02:50:44.019805 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.019773) EVENT_LOG_v1 {"time_micros": 1531363844019767, "job": 1430, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:50:44.019807 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.019785) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:50:44.019827 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1430] Try to delete WAL files size 2762828, prev total WAL file size 2762828, number of live WAL files 2.
>
>2018-07-12 02:50:44.020609 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:50:44.020806 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 02:50:44.020812 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 125870.9 total, 867.3 interval
>Cumulative writes: 153K writes, 714K keys, 153K commit groups, 1.0 writes per commit group, ingest: 1.16 GB, 0.01 MB/s
>Cumulative WAL: 153K writes, 153K syncs, 1.00 writes per sync, written: 1.16 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 876 writes, 3813 keys, 876 commit groups, 1.0 writes per commit group, ingest: 6.17 MB, 0.01 MB/s
>Interval WAL: 876 writes, 876 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.38 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       716    0.077       0      0
>  L1      1/1   14.03 MB   0.0     11.3     1.0     10.3      10.3     -0.0       0.0  10.8     72.5     66.4       159       713    0.223   2561K   312K
> Sum      2/2   15.42 MB   0.0     11.3     1.0     10.3      11.3      1.0       0.0  11.8     53.9     53.9       214      1429    0.150   2561K   312K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.8     56.4     55.9         1         6    0.139     10K   1349
>Uptime(secs): 125870.9 total, 125870.9 interval
>Flush(GB): cumulative 0.957, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.27 GB write, 0.09 MB/s write, 11.27 GB read, 0.09 MB/s read, 214.2 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.38 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       716    0.077       0      0
>  L1      1/1   14.03 MB   0.0     11.3     1.0     10.3      10.3     -0.0       0.0  10.8     72.5     66.4       159       713    0.223   2561K   312K
> Sum      2/2   15.42 MB   0.0     11.3     1.0     10.3      11.3      1.0       0.0  11.8     53.9     53.9       214      1429    0.150   2561K   312K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 125870.9 total, 125870.9 interval
>Flush(GB): cumulative 0.957, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.27 GB write, 0.09 MB/s write, 11.27 GB read, 0.09 MB/s read, 214.2 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 02:50:44.020834 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.020827) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:50:44.020838 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1431] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:50:44.020844 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1430 Base level 0, inputs: [11273(1418KB)], [11271(14MB)]
>
>2018-07-12 02:50:44.020856 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363844020847, "job": 1431, "event": "compaction_started", "files_L0": [11273], "files_L1": [11271], "score": -1, "input_data_size": 16164491}
>2018-07-12 02:50:44.212148 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1431] Generated table #11274: 3170 keys, 15677786 bytes
>2018-07-12 02:50:44.212212 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363844212195, "cf_name": "default", "job": 1431, "event": "table_file_creation", "file_number": 11274, "file_size": 15677786, "table_properties": {"data_size": 15556950, "index_size": 69306, "filter_size": 50603, "raw_key_size": 64841, "raw_average_key_size": 20, "raw_value_size": 15462976, "raw_average_value_size": 4877, "num_data_blocks": 2022, "num_entries": 3170, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:50:44.245335 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1431] Compacted 1@0 + 1@1 files to L1 => 15677786 bytes
>2018-07-12 02:50:44.287212 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.287137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 69.8 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.9) write-amplify(10.8) OK, records in: 3499, records dropped: 329
>
>2018-07-12 02:50:44.287221 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:50:44.287158) EVENT_LOG_v1 {"time_micros": 1531363844287148, "job": 1431, "event": "compaction_finished", "compaction_time_micros": 224464, "output_level": 1, "num_output_files": 1, "total_output_size": 15677786, "num_input_records": 3499, "num_output_records": 3170, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:50:44.287588 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363844287584, "job": 1431, "event": "table_file_deletion", "file_number": 11273}
>2018-07-12 02:50:44.290495 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363844290492, "job": 1431, "event": "table_file_deletion", "file_number": 11271}
>2018-07-12 02:50:44.290608 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:51:12.994127 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:52:06.652786 7f29149e4700  1 mon.magna102@0(leader).log v22374 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22374)
>2018-07-12 02:52:12.994319 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 02:52:46.990535 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11275. Immutable memtables: 0.
>
>2018-07-12 02:52:46.990578 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1432] Syncing log #11272
>2018-07-12 02:52:47.007093 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:52:46.990568) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:52:47.007105 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1432] Flushing memtable with next log file: 11275
>
>2018-07-12 02:52:47.007132 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967007124, "job": 1432, "event": "flush_started", "num_memtables": 1, "num_entries": 698, "num_deletes": 251, "memory_usage": 960400}
>2018-07-12 02:52:47.007135 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1432] Level-0 flush table #11276: started
>2018-07-12 02:52:47.049300 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967049286, "cf_name": "default", "job": 1432, "event": "table_file_creation", "file_number": 11276, "file_size": 913746, "table_properties": {"data_size": 905085, "index_size": 4189, "filter_size": 3552, "raw_key_size": 7912, "raw_average_key_size": 20, "raw_value_size": 896764, "raw_average_value_size": 2317, "num_data_blocks": 128, "num_entries": 387, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 02:52:47.049310 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1432] Level-0 flush table #11276: 913746 bytes OK
>2018-07-12 02:52:47.107650 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.065822) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11276 started
>2018-07-12 02:52:47.107658 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.107614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11276: memtable #1 done
>2018-07-12 02:52:47.107660 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.107628) EVENT_LOG_v1 {"time_micros": 1531363967107622, "job": 1432, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:52:47.107674 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.107640) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:52:47.107681 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1432] Try to delete WAL files size 944949, prev total WAL file size 944949, number of live WAL files 2.
>
>2018-07-12 02:52:47.107995 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:52:47.108049 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.108039) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 02:52:47.108058 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1433] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:52:47.108064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1432 Base level 0, inputs: [11276(892KB)], [11274(14MB)]
>
>2018-07-12 02:52:47.108078 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967108068, "job": 1433, "event": "compaction_started", "files_L0": [11276], "files_L1": [11274], "score": -1, "input_data_size": 16591532}
>2018-07-12 02:52:47.291691 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1433] Generated table #11277: 3047 keys, 14709787 bytes
>2018-07-12 02:52:47.291740 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967291713, "cf_name": "default", "job": 1433, "event": "table_file_creation", "file_number": 11277, "file_size": 14709787, "table_properties": {"data_size": 14595895, "index_size": 65356, "filter_size": 47609, "raw_key_size": 62506, "raw_average_key_size": 20, "raw_value_size": 14506369, "raw_average_value_size": 4760, "num_data_blocks": 1898, "num_entries": 3047, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:52:47.324862 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1433] Compacted 1@0 + 1@1 files to L1 => 14709787 bytes
>2018-07-12 02:52:47.358388 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.358317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.5 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.9, 15.0) out(14.0), read-write-amplify(34.3) write-amplify(16.1) OK, records in: 3557, records dropped: 510
>
>2018-07-12 02:52:47.358397 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:52:47.358339) EVENT_LOG_v1 {"time_micros": 1531363967358329, "job": 1433, "event": "compaction_finished", "compaction_time_micros": 216768, "output_level": 1, "num_output_files": 1, "total_output_size": 14709787, "num_input_records": 3557, "num_output_records": 3047, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:52:47.358661 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967358657, "job": 1433, "event": "table_file_deletion", "file_number": 11276}
>2018-07-12 02:52:47.361589 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531363967361586, "job": 1433, "event": "table_file_deletion", "file_number": 11274}
>2018-07-12 02:53:12.994511 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:53:28.012930 7f29149e4700  1 mon.magna102@0(leader).log v22375 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22375)
>2018-07-12 02:53:58.015034 7f29149e4700  1 mon.magna102@0(leader).log v22376 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22376)
>2018-07-12 02:54:12.994700 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:55:12.994899 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:56:12.995073 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:57:12.998013 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:57:22.815473 7f29149e4700  1 mon.magna102@0(leader).log v22377 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22377)
>2018-07-12 02:58:13.028425 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 02:59:04.115657 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11278. Immutable memtables: 0.
>
>2018-07-12 02:59:04.115701 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1434] Syncing log #11275
>2018-07-12 02:59:04.132081 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.115692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 02:59:04.132093 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1434] Flushing memtable with next log file: 11278
>
>2018-07-12 02:59:04.132119 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344132111, "job": 1434, "event": "flush_started", "num_memtables": 1, "num_entries": 1595, "num_deletes": 250, "memory_usage": 2870432}
>2018-07-12 02:59:04.132122 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1434] Level-0 flush table #11279: started
>2018-07-12 02:59:04.182643 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344182628, "cf_name": "default", "job": 1434, "event": "table_file_creation", "file_number": 11279, "file_size": 1505816, "table_properties": {"data_size": 1493084, "index_size": 6443, "filter_size": 5369, "raw_key_size": 9804, "raw_average_key_size": 21, "raw_value_size": 1482088, "raw_average_value_size": 3207, "num_data_blocks": 202, "num_entries": 462, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 02:59:04.182653 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1434] Level-0 flush table #11279: 1505816 bytes OK
>2018-07-12 02:59:04.241002 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.199167) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11279 started
>2018-07-12 02:59:04.241010 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.240966) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11279: memtable #1 done
>2018-07-12 02:59:04.241013 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.240981) EVENT_LOG_v1 {"time_micros": 1531364344240974, "job": 1434, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 02:59:04.241028 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.240993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 02:59:04.241034 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1434] Try to delete WAL files size 2838561, prev total WAL file size 2838561, number of live WAL files 2.
>
>2018-07-12 02:59:04.241778 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:59:04.241840 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.241832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 02:59:04.241846 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1435] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 02:59:04.241853 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1434 Base level 0, inputs: [11279(1470KB)], [11277(14MB)]
>
>2018-07-12 02:59:04.241866 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344241856, "job": 1435, "event": "compaction_started", "files_L0": [11279], "files_L1": [11277], "score": -1, "input_data_size": 16215603}
>2018-07-12 02:59:04.425045 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1435] Generated table #11280: 3183 keys, 15749220 bytes
>2018-07-12 02:59:04.425095 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344425066, "cf_name": "default", "job": 1435, "event": "table_file_creation", "file_number": 11280, "file_size": 15749220, "table_properties": {"data_size": 15627847, "index_size": 69616, "filter_size": 50830, "raw_key_size": 65105, "raw_average_key_size": 20, "raw_value_size": 15533462, "raw_average_value_size": 4880, "num_data_blocks": 2031, "num_entries": 3183, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 02:59:04.458212 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1435] Compacted 1@0 + 1@1 files to L1 => 15749220 bytes
>2018-07-12 02:59:04.491732 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.491664) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 75.0 rd, 72.8 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.2) write-amplify(10.5) OK, records in: 3509, records dropped: 326
>
>2018-07-12 02:59:04.491741 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-02:59:04.491686) EVENT_LOG_v1 {"time_micros": 1531364344491677, "job": 1435, "event": "compaction_finished", "compaction_time_micros": 216328, "output_level": 1, "num_output_files": 1, "total_output_size": 15749220, "num_input_records": 3509, "num_output_records": 3183, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 02:59:04.492114 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344492110, "job": 1435, "event": "table_file_deletion", "file_number": 11279}
>2018-07-12 02:59:04.494993 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364344494990, "job": 1435, "event": "table_file_deletion", "file_number": 11277}
>2018-07-12 02:59:04.495106 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 02:59:13.044073 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:00:13.073233 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:00:59.211543 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11281. Immutable memtables: 0.
>
>2018-07-12 03:00:59.211587 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1436] Syncing log #11278
>2018-07-12 03:00:59.228129 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.211578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:00:59.228141 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1436] Flushing memtable with next log file: 11281
>
>2018-07-12 03:00:59.228168 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459228160, "job": 1436, "event": "flush_started", "num_memtables": 1, "num_entries": 662, "num_deletes": 251, "memory_usage": 883976}
>2018-07-12 03:00:59.228171 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1436] Level-0 flush table #11282: started
>2018-07-12 03:00:59.270327 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459270313, "cf_name": "default", "job": 1436, "event": "table_file_creation", "file_number": 11282, "file_size": 839812, "table_properties": {"data_size": 831705, "index_size": 3866, "filter_size": 3322, "raw_key_size": 7632, "raw_average_key_size": 20, "raw_value_size": 823832, "raw_average_value_size": 2202, "num_data_blocks": 118, "num_entries": 374, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:00:59.270337 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1436] Level-0 flush table #11282: 839812 bytes OK
>2018-07-12 03:00:59.320311 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.286864) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11282 started
>2018-07-12 03:00:59.320320 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.320276) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11282: memtable #1 done
>2018-07-12 03:00:59.320322 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.320290) EVENT_LOG_v1 {"time_micros": 1531364459320284, "job": 1436, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:00:59.320337 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.320302) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:00:59.320343 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1436] Try to delete WAL files size 868923, prev total WAL file size 868923, number of live WAL files 2.
>
>2018-07-12 03:00:59.320649 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:00:59.320836 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 03:00:59.320842 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 126486.2 total, 615.3 interval
>Cumulative writes: 153K writes, 717K keys, 153K commit groups, 1.0 writes per commit group, ingest: 1.16 GB, 0.01 MB/s
>Cumulative WAL: 153K writes, 153K syncs, 1.00 writes per sync, written: 1.16 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 630 writes, 2955 keys, 630 commit groups, 1.0 writes per commit group, ingest: 4.44 MB, 0.01 MB/s
>Interval WAL: 630 writes, 630 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   820.13 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       719    0.077       0      0
>  L1      1/1   15.02 MB   0.0     11.3     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       160       716    0.223   2571K   313K
> Sum      2/2   15.82 MB   0.0     11.3     1.0     10.4      11.3      1.0       0.0  11.8     53.9     53.9       215      1435    0.150   2571K   313K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  15.2     55.5     55.9         1         6    0.140     10K   1165
>Uptime(secs): 126486.2 total, 126486.2 interval
>Flush(GB): cumulative 0.960, interval 0.003
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.32 GB write, 0.09 MB/s write, 11.32 GB read, 0.09 MB/s read, 215.1 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.8 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   820.13 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       719    0.077       0      0
>  L1      1/1   15.02 MB   0.0     11.3     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       160       716    0.223   2571K   313K
> Sum      2/2   15.82 MB   0.0     11.3     1.0     10.4      11.3      1.0       0.0  11.8     53.9     53.9       215      1435    0.150   2571K   313K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 126486.2 total, 126486.2 interval
>Flush(GB): cumulative 0.960, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.32 GB write, 0.09 MB/s write, 11.32 GB read, 0.09 MB/s read, 215.1 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 03:00:59.320865 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.320858) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:00:59.320872 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1437] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:00:59.320878 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1436 Base level 0, inputs: [11282(820KB)], [11280(15MB)]
>
>2018-07-12 03:00:59.320890 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459320881, "job": 1437, "event": "compaction_started", "files_L0": [11282], "files_L1": [11280], "score": -1, "input_data_size": 16589032}
>2018-07-12 03:00:59.504403 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1437] Generated table #11283: 3049 keys, 14707512 bytes
>2018-07-12 03:00:59.504441 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459504426, "cf_name": "default", "job": 1437, "event": "table_file_creation", "file_number": 11283, "file_size": 14707512, "table_properties": {"data_size": 14593655, "index_size": 65326, "filter_size": 47604, "raw_key_size": 62541, "raw_average_key_size": 20, "raw_value_size": 14504112, "raw_average_value_size": 4757, "num_data_blocks": 1897, "num_entries": 3049, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:00:59.537539 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1437] Compacted 1@0 + 1@1 files to L1 => 14707512 bytes
>2018-07-12 03:00:59.571084 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.571017) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.6 rd, 67.9 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.0) out(14.0), read-write-amplify(37.3) write-amplify(17.5) OK, records in: 3557, records dropped: 508
>
>2018-07-12 03:00:59.571093 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:00:59.571038) EVENT_LOG_v1 {"time_micros": 1531364459571028, "job": 1437, "event": "compaction_finished", "compaction_time_micros": 216631, "output_level": 1, "num_output_files": 1, "total_output_size": 14707512, "num_input_records": 3557, "num_output_records": 3049, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:00:59.571348 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459571344, "job": 1437, "event": "table_file_deletion", "file_number": 11282}
>2018-07-12 03:00:59.574270 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364459574267, "job": 1437, "event": "table_file_deletion", "file_number": 11280}
>2018-07-12 03:01:13.107275 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:02:13.128020 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:03:13.158222 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:04:13.179968 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:05:13.210243 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:06:13.232247 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:07:13.254240 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:07:24.330536 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11284. Immutable memtables: 0.
>
>2018-07-12 03:07:24.330579 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1438] Syncing log #11281
>2018-07-12 03:07:24.346998 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.330569) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:07:24.347009 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1438] Flushing memtable with next log file: 11284
>
>2018-07-12 03:07:24.347035 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844347027, "job": 1438, "event": "flush_started", "num_memtables": 1, "num_entries": 1599, "num_deletes": 250, "memory_usage": 2880664}
>2018-07-12 03:07:24.347039 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1438] Level-0 flush table #11285: started
>2018-07-12 03:07:24.397518 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844397504, "cf_name": "default", "job": 1438, "event": "table_file_creation", "file_number": 11285, "file_size": 1487017, "table_properties": {"data_size": 1474377, "index_size": 6406, "filter_size": 5314, "raw_key_size": 9650, "raw_average_key_size": 21, "raw_value_size": 1463532, "raw_average_value_size": 3216, "num_data_blocks": 201, "num_entries": 455, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 03:07:24.397529 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1438] Level-0 flush table #11285: 1487017 bytes OK
>2018-07-12 03:07:24.447564 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.414072) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11285 started
>2018-07-12 03:07:24.447572 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.447528) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11285: memtable #1 done
>2018-07-12 03:07:24.447575 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.447542) EVENT_LOG_v1 {"time_micros": 1531364844447536, "job": 1438, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:07:24.447590 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.447554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:07:24.447596 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1438] Try to delete WAL files size 2848979, prev total WAL file size 2848979, number of live WAL files 2.
>
>2018-07-12 03:07:24.448311 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:07:24.448361 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.448354) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 03:07:24.448370 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1439] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:07:24.448377 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1438 Base level 0, inputs: [11285(1452KB)], [11283(14MB)]
>
>2018-07-12 03:07:24.448389 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844448380, "job": 1439, "event": "compaction_started", "files_L0": [11285], "files_L1": [11283], "score": -1, "input_data_size": 16194529}
>2018-07-12 03:07:24.639928 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1439] Generated table #11286: 3184 keys, 15755273 bytes
>2018-07-12 03:07:24.639971 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844639944, "cf_name": "default", "job": 1439, "event": "table_file_creation", "file_number": 11286, "file_size": 15755273, "table_properties": {"data_size": 15633847, "index_size": 69648, "filter_size": 50851, "raw_key_size": 65125, "raw_average_key_size": 20, "raw_value_size": 15539425, "raw_average_value_size": 4880, "num_data_blocks": 2032, "num_entries": 3184, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:07:24.681455 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1439] Compacted 1@0 + 1@1 files to L1 => 15755273 bytes
>2018-07-12 03:07:24.714981 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.714912) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.5 rd, 67.6 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.5) write-amplify(10.6) OK, records in: 3504, records dropped: 320
>
>2018-07-12 03:07:24.714991 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:07:24.714933) EVENT_LOG_v1 {"time_micros": 1531364844714924, "job": 1439, "event": "compaction_finished", "compaction_time_micros": 233049, "output_level": 1, "num_output_files": 1, "total_output_size": 15755273, "num_input_records": 3504, "num_output_records": 3184, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:07:24.715351 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844715347, "job": 1439, "event": "table_file_deletion", "file_number": 11285}
>2018-07-12 03:07:24.718213 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364844718210, "job": 1439, "event": "table_file_deletion", "file_number": 11283}
>2018-07-12 03:07:24.718255 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:08:13.278470 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:09:13.309050 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:09:17.443261 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11287. Immutable memtables: 0.
>
>2018-07-12 03:09:17.443307 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1440] Syncing log #11284
>2018-07-12 03:09:17.459783 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.443297) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:09:17.459795 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1440] Flushing memtable with next log file: 11287
>
>2018-07-12 03:09:17.459808 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957459801, "job": 1440, "event": "flush_started", "num_memtables": 1, "num_entries": 655, "num_deletes": 251, "memory_usage": 868768}
>2018-07-12 03:09:17.459812 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1440] Level-0 flush table #11288: started
>2018-07-12 03:09:17.501967 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957501952, "cf_name": "default", "job": 1440, "event": "table_file_creation", "file_number": 11288, "file_size": 825574, "table_properties": {"data_size": 817582, "index_size": 3797, "filter_size": 3276, "raw_key_size": 7590, "raw_average_key_size": 20, "raw_value_size": 809787, "raw_average_value_size": 2176, "num_data_blocks": 116, "num_entries": 372, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:09:17.501978 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1440] Level-0 flush table #11288: 825574 bytes OK
>2018-07-12 03:09:17.551988 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.518542) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11288 started
>2018-07-12 03:09:17.551996 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.551952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11288: memtable #1 done
>2018-07-12 03:09:17.551999 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.551967) EVENT_LOG_v1 {"time_micros": 1531364957551960, "job": 1440, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:09:17.552001 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.551979) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:09:17.552007 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1440] Try to delete WAL files size 854154, prev total WAL file size 854154, number of live WAL files 2.
>
>2018-07-12 03:09:17.552275 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:09:17.552326 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.552320) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:09:17.552336 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1441] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:09:17.552342 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1440 Base level 0, inputs: [11288(806KB)], [11286(15MB)]
>
>2018-07-12 03:09:17.552354 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957552345, "job": 1441, "event": "compaction_started", "files_L0": [11288], "files_L1": [11286], "score": -1, "input_data_size": 16580847}
>2018-07-12 03:09:17.736061 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1441] Generated table #11289: 3048 keys, 14698388 bytes
>2018-07-12 03:09:17.736111 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957736083, "cf_name": "default", "job": 1441, "event": "table_file_creation", "file_number": 11289, "file_size": 14698388, "table_properties": {"data_size": 14584593, "index_size": 65289, "filter_size": 47579, "raw_key_size": 62519, "raw_average_key_size": 20, "raw_value_size": 14495089, "raw_average_value_size": 4755, "num_data_blocks": 1896, "num_entries": 3048, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:09:17.769229 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1441] Compacted 1@0 + 1@1 files to L1 => 14698388 bytes
>2018-07-12 03:09:17.802769 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.802700) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.5 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.0) out(14.0), read-write-amplify(37.9) write-amplify(17.8) OK, records in: 3556, records dropped: 508
>
>2018-07-12 03:09:17.802778 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:09:17.802721) EVENT_LOG_v1 {"time_micros": 1531364957802712, "job": 1441, "event": "compaction_finished", "compaction_time_micros": 216860, "output_level": 1, "num_output_files": 1, "total_output_size": 14698388, "num_input_records": 3556, "num_output_records": 3048, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:09:17.802991 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957802987, "job": 1441, "event": "table_file_deletion", "file_number": 11288}
>2018-07-12 03:09:17.805915 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531364957805911, "job": 1441, "event": "table_file_deletion", "file_number": 11286}
>2018-07-12 03:10:13.333342 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:11:13.363884 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:12:13.384703 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:13:13.414981 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:14:13.437035 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:15:13.467325 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:15:44.536634 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11290. Immutable memtables: 0.
>
>2018-07-12 03:15:44.536676 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1442] Syncing log #11287
>2018-07-12 03:15:44.553107 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.536666) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:15:44.553118 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1442] Flushing memtable with next log file: 11290
>
>2018-07-12 03:15:44.553144 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344553136, "job": 1442, "event": "flush_started", "num_memtables": 1, "num_entries": 1606, "num_deletes": 250, "memory_usage": 2895496}
>2018-07-12 03:15:44.553148 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1442] Level-0 flush table #11291: started
>2018-07-12 03:15:44.603656 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344603642, "cf_name": "default", "job": 1442, "event": "table_file_creation", "file_number": 11291, "file_size": 1494476, "table_properties": {"data_size": 1481778, "index_size": 6439, "filter_size": 5339, "raw_key_size": 9670, "raw_average_key_size": 21, "raw_value_size": 1470895, "raw_average_value_size": 3225, "num_data_blocks": 202, "num_entries": 456, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 03:15:44.603667 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1442] Level-0 flush table #11291: 1494476 bytes OK
>2018-07-12 03:15:44.653688 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.620176) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11291 started
>2018-07-12 03:15:44.653696 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.653652) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11291: memtable #1 done
>2018-07-12 03:15:44.653698 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.653666) EVENT_LOG_v1 {"time_micros": 1531365344653660, "job": 1442, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:15:44.653701 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.653678) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:15:44.653721 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1442] Try to delete WAL files size 2863748, prev total WAL file size 2863748, number of live WAL files 2.
>
>2018-07-12 03:15:44.654442 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:15:44.654627 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 03:15:44.654634 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 127371.5 total, 885.3 interval
>Cumulative writes: 154K writes, 721K keys, 154K commit groups, 1.0 writes per commit group, ingest: 1.17 GB, 0.01 MB/s
>Cumulative WAL: 154K writes, 154K syncs, 1.00 writes per sync, written: 1.17 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 890 writes, 3860 keys, 890 commit groups, 1.0 writes per commit group, ingest: 6.26 MB, 0.01 MB/s
>Interval WAL: 890 writes, 890 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.43 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       722    0.077       0      0
>  L1      1/1   14.02 MB   0.0     11.4     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       160       719    0.223   2582K   315K
> Sum      2/2   15.44 MB   0.0     11.4     1.0     10.4      11.4      1.0       0.0  11.8     53.9     53.9       216      1441    0.150   2582K   315K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.9     54.8     54.3         1         6    0.143     10K   1336
>Uptime(secs): 127371.5 total, 127371.5 interval
>Flush(GB): cumulative 0.963, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.36 GB write, 0.09 MB/s write, 11.36 GB read, 0.09 MB/s read, 215.9 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.43 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        55       722    0.077       0      0
>  L1      1/1   14.02 MB   0.0     11.4     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       160       719    0.223   2582K   315K
> Sum      2/2   15.44 MB   0.0     11.4     1.0     10.4      11.4      1.0       0.0  11.8     53.9     53.9       216      1441    0.150   2582K   315K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 127371.5 total, 127371.5 interval
>Flush(GB): cumulative 0.963, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.36 GB write, 0.09 MB/s write, 11.36 GB read, 0.09 MB/s read, 215.9 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 03:15:44.654654 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.654648) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 03:15:44.654659 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1443] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:15:44.654664 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1442 Base level 0, inputs: [11291(1459KB)], [11289(14MB)]
>
>2018-07-12 03:15:44.654677 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344654668, "job": 1443, "event": "compaction_started", "files_L0": [11291], "files_L1": [11289], "score": -1, "input_data_size": 16192864}
>2018-07-12 03:15:44.846053 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1443] Generated table #11292: 3185 keys, 15760385 bytes
>2018-07-12 03:15:44.846102 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344846075, "cf_name": "default", "job": 1443, "event": "table_file_creation", "file_number": 11292, "file_size": 15760385, "table_properties": {"data_size": 15638906, "index_size": 69680, "filter_size": 50872, "raw_key_size": 65145, "raw_average_key_size": 20, "raw_value_size": 15544447, "raw_average_value_size": 4880, "num_data_blocks": 2033, "num_entries": 3185, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:15:44.879219 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1443] Compacted 1@0 + 1@1 files to L1 => 15760385 bytes
>2018-07-12 03:15:44.921085 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.921019) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.1 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.4) write-amplify(10.5) OK, records in: 3504, records dropped: 319
>
>2018-07-12 03:15:44.921093 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:15:44.921040) EVENT_LOG_v1 {"time_micros": 1531365344921030, "job": 1443, "event": "compaction_finished", "compaction_time_micros": 224525, "output_level": 1, "num_output_files": 1, "total_output_size": 15760385, "num_input_records": 3504, "num_output_records": 3185, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:15:44.921469 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344921464, "job": 1443, "event": "table_file_deletion", "file_number": 11291}
>2018-07-12 03:15:44.924325 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365344924310, "job": 1443, "event": "table_file_deletion", "file_number": 11289}
>2018-07-12 03:15:44.924355 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:16:13.491414 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:17:13.513493 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:17:35.623858 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11293. Immutable memtables: 0.
>
>2018-07-12 03:17:35.623902 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1444] Syncing log #11290
>2018-07-12 03:17:35.640418 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.623893) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:17:35.640429 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1444] Flushing memtable with next log file: 11293
>
>2018-07-12 03:17:35.640442 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365455640435, "job": 1444, "event": "flush_started", "num_memtables": 1, "num_entries": 648, "num_deletes": 251, "memory_usage": 854024}
>2018-07-12 03:17:35.640446 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1444] Level-0 flush table #11294: started
>2018-07-12 03:17:35.690972 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365455690958, "cf_name": "default", "job": 1444, "event": "table_file_creation", "file_number": 11294, "file_size": 811346, "table_properties": {"data_size": 803464, "index_size": 3733, "filter_size": 3230, "raw_key_size": 7548, "raw_average_key_size": 20, "raw_value_size": 795742, "raw_average_value_size": 2150, "num_data_blocks": 114, "num_entries": 370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:17:35.690982 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1444] Level-0 flush table #11294: 811346 bytes OK
>2018-07-12 03:17:35.749354 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.707531) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11294 started
>2018-07-12 03:17:35.749362 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.749318) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11294: memtable #1 done
>2018-07-12 03:17:35.749365 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.749333) EVENT_LOG_v1 {"time_micros": 1531365455749326, "job": 1444, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:17:35.749367 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.749344) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:17:35.749374 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1444] Try to delete WAL files size 839385, prev total WAL file size 839385, number of live WAL files 2.
>
>2018-07-12 03:17:35.749626 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:17:35.749681 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:17:35.749674) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:17:35.749690 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1445] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:17:35.749696 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1444 Base level 0, inputs: [11294(792KB)], [11292(15MB)]
>
>2018-07-12 03:17:35.749709 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365455749700, "job": 1445, "event": "compaction_started", "files_L0": [11294], "files_L1": [11292], "score": -1, "input_data_size": 16571731}
>2018-07-12 03:17:35.933388 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1445] Generated table #11295: 3047 keys, 14689399 bytes
>2018-07-12 03:17:35.933420 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365455933405, "cf_name": "default", "job": 1445, "event": "table_file_creation", "file_number": 11295, "file_size": 14689399, "table_properties": {"data_size": 14575668, "index_size": 65254, "filter_size": 47550, "raw_key_size": 62497, "raw_average_key_size": 20, "raw_value_size": 14486203, "raw_average_value_size": 4754, "num_data_blocks": 1895, "num_entries": 3047, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:17:35.966459 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1445] Compacted 1@0 + 1@1 files to L1 => 14689399 bytes
>2018-07-12 03:17:36.008446 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:17:36.008374) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.5 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(0.8, 15.0) out(14.0), read-write-amplify(38.5) write-amplify(18.1) OK, records in: 3555, records dropped: 508
>
>2018-07-12 03:17:36.008455 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:17:36.008395) EVENT_LOG_v1 {"time_micros": 1531365456008386, "job": 1445, "event": "compaction_finished", "compaction_time_micros": 216734, "output_level": 1, "num_output_files": 1, "total_output_size": 14689399, "num_input_records": 3555, "num_output_records": 3047, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:17:36.008672 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365456008668, "job": 1445, "event": "table_file_deletion", "file_number": 11294}
>2018-07-12 03:17:36.011611 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365456011608, "job": 1445, "event": "table_file_deletion", "file_number": 11292}
>2018-07-12 03:18:13.537785 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:19:13.568101 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:20:13.597359 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:21:13.619748 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:22:13.666796 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:23:13.672085 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:24:04.760616 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11296. Immutable memtables: 0.
>
>2018-07-12 03:24:04.760658 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1446] Syncing log #11293
>2018-07-12 03:24:04.777087 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.760649) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:24:04.777099 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1446] Flushing memtable with next log file: 11296
>
>2018-07-12 03:24:04.777112 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365844777105, "job": 1446, "event": "flush_started", "num_memtables": 1, "num_entries": 1620, "num_deletes": 250, "memory_usage": 2912760}
>2018-07-12 03:24:04.777115 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1446] Level-0 flush table #11297: started
>2018-07-12 03:24:04.827611 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365844827596, "cf_name": "default", "job": 1446, "event": "table_file_creation", "file_number": 11297, "file_size": 1503787, "table_properties": {"data_size": 1491030, "index_size": 6471, "filter_size": 5366, "raw_key_size": 9780, "raw_average_key_size": 21, "raw_value_size": 1480030, "raw_average_value_size": 3210, "num_data_blocks": 203, "num_entries": 461, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 03:24:04.827621 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1446] Level-0 flush table #11297: 1503787 bytes OK
>2018-07-12 03:24:04.877684 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.844162) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11297 started
>2018-07-12 03:24:04.877692 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.877648) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11297: memtable #1 done
>2018-07-12 03:24:04.877695 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.877663) EVENT_LOG_v1 {"time_micros": 1531365844877656, "job": 1446, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:24:04.877697 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.877674) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:24:04.877704 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1446] Try to delete WAL files size 2880469, prev total WAL file size 2880469, number of live WAL files 2.
>
>2018-07-12 03:24:04.878405 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:24:04.878458 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:24:04.878451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 03:24:04.878467 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1447] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:24:04.878473 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1446 Base level 0, inputs: [11297(1468KB)], [11295(14MB)]
>
>2018-07-12 03:24:04.878486 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365844878476, "job": 1447, "event": "compaction_started", "files_L0": [11297], "files_L1": [11295], "score": -1, "input_data_size": 16193186}
>2018-07-12 03:24:05.070055 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1447] Generated table #11298: 3188 keys, 15767423 bytes
>2018-07-12 03:24:05.070099 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365845070072, "cf_name": "default", "job": 1447, "event": "table_file_creation", "file_number": 11298, "file_size": 15767423, "table_properties": {"data_size": 15645886, "index_size": 69715, "filter_size": 50895, "raw_key_size": 65201, "raw_average_key_size": 20, "raw_value_size": 15551362, "raw_average_value_size": 4878, "num_data_blocks": 2034, "num_entries": 3188, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:24:05.103256 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1447] Compacted 1@0 + 1@1 files to L1 => 15767423 bytes
>2018-07-12 03:24:05.136767 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:24:05.136700) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.3) write-amplify(10.5) OK, records in: 3508, records dropped: 320
>
>2018-07-12 03:24:05.136776 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:24:05.136722) EVENT_LOG_v1 {"time_micros": 1531365845136712, "job": 1447, "event": "compaction_finished", "compaction_time_micros": 224753, "output_level": 1, "num_output_files": 1, "total_output_size": 15767423, "num_input_records": 3508, "num_output_records": 3188, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:24:05.137105 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365845137099, "job": 1447, "event": "table_file_deletion", "file_number": 11297}
>2018-07-12 03:24:05.139947 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365845139944, "job": 1447, "event": "table_file_deletion", "file_number": 11295}
>2018-07-12 03:24:05.140001 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:24:13.696240 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:25:13.718251 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:25:51.829710 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11299. Immutable memtables: 0.
>
>2018-07-12 03:25:51.829753 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1448] Syncing log #11296
>2018-07-12 03:25:51.846267 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.829744) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:25:51.846279 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1448] Flushing memtable with next log file: 11299
>
>2018-07-12 03:25:51.846305 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365951846297, "job": 1448, "event": "flush_started", "num_memtables": 1, "num_entries": 634, "num_deletes": 251, "memory_usage": 824392}
>2018-07-12 03:25:51.846309 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1448] Level-0 flush table #11300: started
>2018-07-12 03:25:51.888471 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365951888457, "cf_name": "default", "job": 1448, "event": "table_file_creation", "file_number": 11300, "file_size": 782875, "table_properties": {"data_size": 775219, "index_size": 3599, "filter_size": 3138, "raw_key_size": 7464, "raw_average_key_size": 20, "raw_value_size": 767652, "raw_average_value_size": 2097, "num_data_blocks": 110, "num_entries": 366, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:25:51.888481 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1448] Level-0 flush table #11300: 782875 bytes OK
>2018-07-12 03:25:51.938515 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.904999) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11300 started
>2018-07-12 03:25:51.938523 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.938479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11300: memtable #1 done
>2018-07-12 03:25:51.938525 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.938493) EVENT_LOG_v1 {"time_micros": 1531365951938487, "job": 1448, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:25:51.938540 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.938505) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:25:51.938547 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1448] Try to delete WAL files size 809847, prev total WAL file size 809847, number of live WAL files 2.
>
>2018-07-12 03:25:51.938840 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:25:51.939024 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 03:25:51.939031 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 127978.8 total, 607.3 interval
>Cumulative writes: 155K writes, 724K keys, 155K commit groups, 1.0 writes per commit group, ingest: 1.17 GB, 0.01 MB/s
>Cumulative WAL: 155K writes, 155K syncs, 1.00 writes per sync, written: 1.17 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 616 writes, 2902 keys, 616 commit groups, 1.0 writes per commit group, ingest: 4.32 MB, 0.01 MB/s
>Interval WAL: 616 writes, 616 syncs, 1.00 writes per sync, written: 0.00 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   764.53 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        56       725    0.077       0      0
>  L1      1/1   15.04 MB   0.0     11.4     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       161       722    0.223   2592K   316K
> Sum      2/2   15.78 MB   0.0     11.4     1.0     10.4      11.4      1.0       0.0  11.8     53.9     53.9       217      1447    0.150   2592K   316K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  15.9     54.4     54.8         1         6    0.143     10K   1147
>Uptime(secs): 127978.8 total, 127978.8 interval
>Flush(GB): cumulative 0.966, interval 0.003
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.41 GB write, 0.09 MB/s write, 11.41 GB read, 0.09 MB/s read, 216.8 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1   764.53 KB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        56       725    0.077       0      0
>  L1      1/1   15.04 MB   0.0     11.4     1.0     10.4      10.4     -0.0       0.0  10.8     72.5     66.4       161       722    0.223   2592K   316K
> Sum      2/2   15.78 MB   0.0     11.4     1.0     10.4      11.4      1.0       0.0  11.8     53.9     53.9       217      1447    0.150   2592K   316K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 127978.8 total, 127978.8 interval
>Flush(GB): cumulative 0.966, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.41 GB write, 0.09 MB/s write, 11.41 GB read, 0.09 MB/s read, 216.8 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 03:25:51.939052 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:25:51.939045) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:25:51.939059 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1449] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:25:51.939064 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1448 Base level 0, inputs: [11300(764KB)], [11298(15MB)]
>
>2018-07-12 03:25:51.939077 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365951939067, "job": 1449, "event": "compaction_started", "files_L0": [11300], "files_L1": [11298], "score": -1, "input_data_size": 16550298}
>2018-07-12 03:25:52.122523 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1449] Generated table #11301: 3046 keys, 14670180 bytes
>2018-07-12 03:25:52.122571 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365952122543, "cf_name": "default", "job": 1449, "event": "table_file_creation", "file_number": 11301, "file_size": 14670180, "table_properties": {"data_size": 14556610, "index_size": 65154, "filter_size": 47489, "raw_key_size": 62469, "raw_average_key_size": 20, "raw_value_size": 14467232, "raw_average_value_size": 4749, "num_data_blocks": 1892, "num_entries": 3046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:25:52.155743 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1449] Compacted 1@0 + 1@1 files to L1 => 14670180 bytes
>2018-07-12 03:25:52.189261 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:25:52.189170) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 76.4 rd, 67.7 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.0) out(14.0), read-write-amplify(39.9) write-amplify(18.7) OK, records in: 3554, records dropped: 508
>
>2018-07-12 03:25:52.189270 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:25:52.189211) EVENT_LOG_v1 {"time_micros": 1531365952189201, "job": 1449, "event": "compaction_finished", "compaction_time_micros": 216650, "output_level": 1, "num_output_files": 1, "total_output_size": 14670180, "num_input_records": 3554, "num_output_records": 3046, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:25:52.189497 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365952189479, "job": 1449, "event": "table_file_deletion", "file_number": 11300}
>2018-07-12 03:25:52.192461 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531365952192458, "job": 1449, "event": "table_file_deletion", "file_number": 11298}
>2018-07-12 03:26:13.750615 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:27:13.772741 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:28:13.801915 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:29:13.823894 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:30:13.887823 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:31:13.887996 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:32:13.898815 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:32:24.975269 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11302. Immutable memtables: 0.
>
>2018-07-12 03:32:24.975313 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1450] Syncing log #11299
>2018-07-12 03:32:24.991661 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:32:24.975304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:32:24.991672 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1450] Flushing memtable with next log file: 11302
>
>2018-07-12 03:32:24.991686 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366344991678, "job": 1450, "event": "flush_started", "num_memtables": 1, "num_entries": 1627, "num_deletes": 250, "memory_usage": 2940704}
>2018-07-12 03:32:24.991689 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1450] Level-0 flush table #11303: started
>2018-07-12 03:32:25.042235 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366345042219, "cf_name": "default", "job": 1450, "event": "table_file_creation", "file_number": 11303, "file_size": 1516844, "table_properties": {"data_size": 1503979, "index_size": 6535, "filter_size": 5410, "raw_key_size": 9730, "raw_average_key_size": 21, "raw_value_size": 1492984, "raw_average_value_size": 3252, "num_data_blocks": 205, "num_entries": 459, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 03:32:25.042245 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1450] Level-0 flush table #11303: 1516844 bytes OK
>2018-07-12 03:32:25.092227 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.058725) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11303 started
>2018-07-12 03:32:25.092236 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.092191) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11303: memtable #1 done
>2018-07-12 03:32:25.092238 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.092206) EVENT_LOG_v1 {"time_micros": 1531366345092199, "job": 1450, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:32:25.092240 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.092218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:32:25.092247 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1450] Try to delete WAL files size 2908055, prev total WAL file size 2908055, number of live WAL files 2.
>
>2018-07-12 03:32:25.092914 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:32:25.092967 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.092960) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 03:32:25.092976 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1451] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:32:25.092982 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1450 Base level 0, inputs: [11303(1481KB)], [11301(13MB)]
>
>2018-07-12 03:32:25.092995 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366345092985, "job": 1451, "event": "compaction_started", "files_L0": [11303], "files_L1": [11301], "score": -1, "input_data_size": 16187024}
>2018-07-12 03:32:25.284615 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1451] Generated table #11304: 3189 keys, 15774883 bytes
>2018-07-12 03:32:25.284665 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366345284637, "cf_name": "default", "job": 1451, "event": "table_file_creation", "file_number": 11304, "file_size": 15774883, "table_properties": {"data_size": 15653286, "index_size": 69750, "filter_size": 50920, "raw_key_size": 65221, "raw_average_key_size": 20, "raw_value_size": 15558725, "raw_average_value_size": 4878, "num_data_blocks": 2035, "num_entries": 3189, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:32:25.317791 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1451] Compacted 1@0 + 1@1 files to L1 => 15774883 bytes
>2018-07-12 03:32:25.351327 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.351257) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 72.0 rd, 70.2 wr, level 1, files in(1, 1) out(1) MB in(1.4, 14.0) out(15.0), read-write-amplify(21.1) write-amplify(10.4) OK, records in: 3505, records dropped: 316
>
>2018-07-12 03:32:25.351336 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:32:25.351278) EVENT_LOG_v1 {"time_micros": 1531366345351268, "job": 1451, "event": "compaction_finished", "compaction_time_micros": 224781, "output_level": 1, "num_output_files": 1, "total_output_size": 15774883, "num_input_records": 3505, "num_output_records": 3189, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:32:25.351680 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366345351676, "job": 1451, "event": "table_file_deletion", "file_number": 11303}
>2018-07-12 03:32:25.354556 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366345354553, "job": 1451, "event": "table_file_deletion", "file_number": 11301}
>2018-07-12 03:32:25.354611 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:33:13.931379 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:34:10.053518 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11305. Immutable memtables: 0.
>
>2018-07-12 03:34:10.053562 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1452] Syncing log #11302
>2018-07-12 03:34:10.070159 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.053552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:34:10.070171 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1452] Flushing memtable with next log file: 11305
>
>2018-07-12 03:34:10.070205 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450070196, "job": 1452, "event": "flush_started", "num_memtables": 1, "num_entries": 627, "num_deletes": 251, "memory_usage": 809504}
>2018-07-12 03:34:10.070208 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1452] Level-0 flush table #11306: started
>2018-07-12 03:34:10.112423 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450112409, "cf_name": "default", "job": 1452, "event": "table_file_creation", "file_number": 11306, "file_size": 768646, "table_properties": {"data_size": 761100, "index_size": 3535, "filter_size": 3092, "raw_key_size": 7422, "raw_average_key_size": 20, "raw_value_size": 753607, "raw_average_value_size": 2070, "num_data_blocks": 108, "num_entries": 364, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:34:10.112434 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1452] Level-0 flush table #11306: 768646 bytes OK
>2018-07-12 03:34:10.162399 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.129000) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11306 started
>2018-07-12 03:34:10.162407 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.162363) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11306: memtable #1 done
>2018-07-12 03:34:10.162409 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.162377) EVENT_LOG_v1 {"time_micros": 1531366450162371, "job": 1452, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:34:10.162411 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.162389) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:34:10.162418 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1452] Try to delete WAL files size 795078, prev total WAL file size 795078, number of live WAL files 2.
>
>2018-07-12 03:34:10.162650 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:34:10.162701 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.162694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:34:10.162710 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1453] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:34:10.162717 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1452 Base level 0, inputs: [11306(750KB)], [11304(15MB)]
>
>2018-07-12 03:34:10.162730 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450162720, "job": 1453, "event": "compaction_started", "files_L0": [11306], "files_L1": [11304], "score": -1, "input_data_size": 16543529}
>2018-07-12 03:34:10.338113 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1453] Generated table #11307: 3045 keys, 14663404 bytes
>2018-07-12 03:34:10.338162 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450338134, "cf_name": "default", "job": 1453, "event": "table_file_creation", "file_number": 11307, "file_size": 14663404, "table_properties": {"data_size": 14549889, "index_size": 65120, "filter_size": 47468, "raw_key_size": 62447, "raw_average_key_size": 20, "raw_value_size": 14460550, "raw_average_value_size": 4748, "num_data_blocks": 1891, "num_entries": 3045, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:34:10.371295 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1453] Compacted 1@0 + 1@1 files to L1 => 14663404 bytes
>2018-07-12 03:34:10.404822 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.404754) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.3 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.0) out(14.0), read-write-amplify(40.6) write-amplify(19.1) OK, records in: 3553, records dropped: 508
>
>2018-07-12 03:34:10.404831 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:34:10.404776) EVENT_LOG_v1 {"time_micros": 1531366450404767, "job": 1453, "event": "compaction_finished", "compaction_time_micros": 208549, "output_level": 1, "num_output_files": 1, "total_output_size": 14663404, "num_input_records": 3553, "num_output_records": 3045, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:34:10.405034 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450405030, "job": 1453, "event": "table_file_deletion", "file_number": 11306}
>2018-07-12 03:34:10.407906 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366450407903, "job": 1453, "event": "table_file_deletion", "file_number": 11304}
>2018-07-12 03:34:13.955355 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:35:13.985864 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:35:17.111222 7f29149e4700  1 mon.magna102@0(leader).log v22378 check_sub sending message to client.24106 10.8.128.102:0/2030830768 with 0 entries (version 22378)
>2018-07-12 03:36:14.006809 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:37:14.037204 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:38:14.067437 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:39:14.089649 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:40:14.111662 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB
>2018-07-12 03:40:45.197880 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11308. Immutable memtables: 0.
>
>2018-07-12 03:40:45.197923 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1454] Syncing log #11305
>2018-07-12 03:40:45.214317 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.197914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:40:45.214329 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1454] Flushing memtable with next log file: 11308
>
>2018-07-12 03:40:45.214355 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845214335, "job": 1454, "event": "flush_started", "num_memtables": 1, "num_entries": 1642, "num_deletes": 250, "memory_usage": 2971240}
>2018-07-12 03:40:45.214359 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1454] Level-0 flush table #11309: started
>2018-07-12 03:40:45.273211 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845273197, "cf_name": "default", "job": 1454, "event": "table_file_creation", "file_number": 11309, "file_size": 1539842, "table_properties": {"data_size": 1526849, "index_size": 6596, "filter_size": 5477, "raw_key_size": 9862, "raw_average_key_size": 21, "raw_value_size": 1515703, "raw_average_value_size": 3259, "num_data_blocks": 207, "num_entries": 465, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "250", "kMergeOperands": "0"}}
>2018-07-12 03:40:45.273220 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1454] Level-0 flush table #11309: 1539842 bytes OK
>2018-07-12 03:40:45.331528 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.289721) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11309 started
>2018-07-12 03:40:45.331536 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.331492) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11309: memtable #1 done
>2018-07-12 03:40:45.331538 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.331506) EVENT_LOG_v1 {"time_micros": 1531366845331500, "job": 1454, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:40:45.331540 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.331518) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:40:45.331559 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1454] Try to delete WAL files size 2938414, prev total WAL file size 2938414, number of live WAL files 2.
>
>2018-07-12 03:40:45.332301 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:40:45.332484 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS -------
>2018-07-12 03:40:45.332490 7f29131e1700  3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] 
>** DB Stats **
>Uptime(secs): 128872.2 total, 893.4 interval
>Cumulative writes: 156K writes, 727K keys, 156K commit groups, 1.0 writes per commit group, ingest: 1.18 GB, 0.01 MB/s
>Cumulative WAL: 156K writes, 156K syncs, 1.00 writes per sync, written: 1.18 GB, 0.01 MB/s
>Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
>Interval writes: 900 writes, 3896 keys, 900 commit groups, 1.0 writes per commit group, ingest: 6.33 MB, 0.01 MB/s
>Interval WAL: 900 writes, 900 syncs, 1.00 writes per sync, written: 0.01 MB, 0.01 MB/s
>Interval stall: 00:00:0.000 H:M:S, 0.0 percent
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.47 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        56       728    0.077       0      0
>  L1      1/1   13.98 MB   0.0     11.5     1.0     10.5      10.5     -0.0       0.0  10.8     72.5     66.4       162       725    0.223   2603K   317K
> Sum      2/2   15.45 MB   0.0     11.5     1.0     10.5      11.5      1.0       0.0  11.8     53.9     53.9       218      1453    0.150   2603K   317K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  12.8     55.2     54.8         1         6    0.142     10K   1332
>Uptime(secs): 128872.2 total, 128872.2 interval
>Flush(GB): cumulative 0.970, interval 0.004
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.45 GB write, 0.09 MB/s write, 11.45 GB read, 0.09 MB/s read, 217.6 seconds
>Interval compaction: 0.05 GB write, 0.00 MB/s write, 0.05 GB read, 0.00 MB/s read, 0.9 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>** Compaction Stats [default] **
>Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
>----------------------------------------------------------------------------------------------------------------------------------------------------------
>  L0      1/1    1.47 MB   0.0      0.0     0.0      0.0       1.0      1.0       0.0   1.0      0.0     17.8        56       728    0.077       0      0
>  L1      1/1   13.98 MB   0.0     11.5     1.0     10.5      10.5     -0.0       0.0  10.8     72.5     66.4       162       725    0.223   2603K   317K
> Sum      2/2   15.45 MB   0.0     11.5     1.0     10.5      11.5      1.0       0.0  11.8     53.9     53.9       218      1453    0.150   2603K   317K
> Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
>Uptime(secs): 128872.2 total, 128872.2 interval
>Flush(GB): cumulative 0.970, interval 0.000
>AddFile(GB): cumulative 0.000, interval 0.000
>AddFile(Total Files): cumulative 0, interval 0
>AddFile(L0 Files): cumulative 0, interval 0
>AddFile(Keys): cumulative 0, interval 0
>Cumulative compaction: 11.45 GB write, 0.09 MB/s write, 11.45 GB read, 0.09 MB/s read, 217.6 seconds
>Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
>Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
>
>** File Read Latency Histogram By Level [default] **
>
>2018-07-12 03:40:45.332512 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.332506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'mgrstat .. 'mgrstat; will stop at (end)
>
>2018-07-12 03:40:45.332517 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1455] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:40:45.332522 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1454 Base level 0, inputs: [11309(1503KB)], [11307(13MB)]
>
>2018-07-12 03:40:45.332535 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845332525, "job": 1455, "event": "compaction_started", "files_L0": [11309], "files_L1": [11307], "score": -1, "input_data_size": 16203246}
>2018-07-12 03:40:45.523933 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1455] Generated table #11310: 3193 keys, 15797884 bytes
>2018-07-12 03:40:45.523983 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845523955, "cf_name": "default", "job": 1455, "event": "table_file_creation", "file_number": 11310, "file_size": 15797884, "table_properties": {"data_size": 15676112, "index_size": 69852, "filter_size": 50993, "raw_key_size": 65302, "raw_average_key_size": 20, "raw_value_size": 15581428, "raw_average_value_size": 4879, "num_data_blocks": 2038, "num_entries": 3193, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:40:45.565457 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1455] Compacted 1@0 + 1@1 files to L1 => 15797884 bytes
>2018-07-12 03:40:45.607306 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.607237) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.06, MB/sec: 69.6 rd, 67.8 wr, level 1, files in(1, 1) out(1) MB in(1.5, 14.0) out(15.1), read-write-amplify(20.8) write-amplify(10.3) OK, records in: 3510, records dropped: 317
>
>2018-07-12 03:40:45.607315 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:40:45.607259) EVENT_LOG_v1 {"time_micros": 1531366845607249, "job": 1455, "event": "compaction_finished", "compaction_time_micros": 232905, "output_level": 1, "num_output_files": 1, "total_output_size": 15797884, "num_input_records": 3510, "num_output_records": 3193, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:40:45.607690 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845607686, "job": 1455, "event": "table_file_deletion", "file_number": 11309}
>2018-07-12 03:40:45.610595 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366845610592, "job": 1455, "event": "table_file_deletion", "file_number": 11307}
>2018-07-12 03:40:45.610708 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:41:14.135985 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:42:14.166277 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8799 MB, avail 861 GB
>2018-07-12 03:42:26.284426 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #11311. Immutable memtables: 0.
>
>2018-07-12 03:42:26.284471 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:49] [JOB 1456] Syncing log #11308
>2018-07-12 03:42:26.300882 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.284462) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1
>2018-07-12 03:42:26.300893 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 1456] Flushing memtable with next log file: 11311
>
>2018-07-12 03:42:26.300907 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946300900, "job": 1456, "event": "flush_started", "num_memtables": 1, "num_entries": 613, "num_deletes": 251, "memory_usage": 779656}
>2018-07-12 03:42:26.300910 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 1456] Level-0 flush table #11312: started
>2018-07-12 03:42:26.343070 7f29139e2700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946343056, "cf_name": "default", "job": 1456, "event": "table_file_creation", "file_number": 11312, "file_size": 740175, "table_properties": {"data_size": 732856, "index_size": 3400, "filter_size": 3000, "raw_key_size": 7338, "raw_average_key_size": 20, "raw_value_size": 725517, "raw_average_value_size": 2015, "num_data_blocks": 104, "num_entries": 360, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "251", "kMergeOperands": "0"}}
>2018-07-12 03:42:26.343080 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 1456] Level-0 flush table #11312: 740175 bytes OK
>2018-07-12 03:42:26.401484 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.359646) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #11312 started
>2018-07-12 03:42:26.401492 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.401448) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #11312: memtable #1 done
>2018-07-12 03:42:26.401495 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.401463) EVENT_LOG_v1 {"time_micros": 1531366946401457, "job": 1456, "event": "flush_finished", "lsm_state": [1, 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>2018-07-12 03:42:26.401497 7f29139e2700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.401474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 1 0 0 0 0 0] max score 0.25
>
>2018-07-12 03:42:26.401503 7f29139e2700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 1456] Try to delete WAL files size 765540, prev total WAL file size 765540, number of live WAL files 2.
>
>2018-07-12 03:42:26.401741 7f29141e3700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:839] [default] Manual compaction starting
>2018-07-12 03:42:26.401795 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.401788) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1392] [default] Manual compaction from level-0 to level-1 from 'paxos .. 'paxos; will stop at (end)
>
>2018-07-12 03:42:26.401804 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 1457] Compacting 1@0 + 1@1 files to L1, score -1.00
>2018-07-12 03:42:26.401810 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 1456 Base level 0, inputs: [11312(722KB)], [11310(15MB)]
>
>2018-07-12 03:42:26.401823 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946401813, "job": 1457, "event": "compaction_started", "files_L0": [11312], "files_L1": [11310], "score": -1, "input_data_size": 16538059}
>2018-07-12 03:42:26.577203 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 1457] Generated table #11313: 3045 keys, 14664419 bytes
>2018-07-12 03:42:26.577234 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946577219, "cf_name": "default", "job": 1457, "event": "table_file_creation", "file_number": 11313, "file_size": 14664419, "table_properties": {"data_size": 14550905, "index_size": 65117, "filter_size": 47470, "raw_key_size": 62444, "raw_average_key_size": 20, "raw_value_size": 14461571, "raw_average_value_size": 4749, "num_data_blocks": 1891, "num_entries": 3045, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}}
>2018-07-12 03:42:26.610375 7f29131e1700  4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 1457] Compacted 1@0 + 1@1 files to L1 => 14664419 bytes
>2018-07-12 03:42:26.652242 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.652161) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0 0 0 0] max score 0.05, MB/sec: 79.3 rd, 70.3 wr, level 1, files in(1, 1) out(1) MB in(0.7, 15.1) out(14.0), read-write-amplify(42.2) write-amplify(19.8) OK, records in: 3553, records dropped: 508
>
>2018-07-12 03:42:26.652251 7f29131e1700  4 rocksdb: (Original Log Time 2018/07/12-03:42:26.652193) EVENT_LOG_v1 {"time_micros": 1531366946652173, "job": 1457, "event": "compaction_finished", "compaction_time_micros": 208535, "output_level": 1, "num_output_files": 1, "total_output_size": 14664419, "num_input_records": 3553, "num_output_records": 3045, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
>2018-07-12 03:42:26.652447 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946652443, "job": 1457, "event": "table_file_deletion", "file_number": 11312}
>2018-07-12 03:42:26.655403 7f29131e1700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531366946655400, "job": 1457, "event": "table_file_deletion", "file_number": 11310}
>2018-07-12 03:43:14.190373 7f291b9f2700  0 mon.magna102@0(leader).data_health(17) update_stats avail 93% total 916 GB, used 8798 MB, avail 861 GB

Actions: View

Attachments on bug 1600138: 1458119 | 1458772 | 1458773