Attachment 1026671 Details for Bug 1222509 – mon log

mon log

file_1222509.txt (text/plain), 82.98 KB, created by Harish NV Rao on 2015-05-18 11:53:05 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Harish NV Rao

Created: 2015-05-18 11:53:05 UTC

Size: 82.98 KB

patch

obsolete

>2015-05-18 06:20:01.644938 7f09890a87c0  0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992
>2015-05-18 06:20:01.759779 7f09890a87c0  0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
>2015-05-18 06:20:01.759988 7f09890a87c0  1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
>2015-05-18 06:20:01.760303 7f09890a87c0  1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1
>2015-05-18 06:20:01.760314 7f09890a87c0  1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap
>2015-05-18 06:20:01.760927 7f09890a87c0  0 mon.Mon@-1(probing).mds e1 print_map
>epoch	1
>flags	0
>created	0.000000
>modified	2015-05-06 15:39:42.874456
>tableserver	0
>root	0
>session_timeout	0
>session_autoclose	0
>max_file_size	0
>last_failure	0
>last_failure_osd_epoch	0
>compat	compat={},rocompat={},incompat={}
>max_mds	0
>in	
>up	{}
>failed	
>stopped	
>data_pools	
>metadata_pool	0
>inline_data	disabled
>
>2015-05-18 06:20:01.761127 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>2015-05-18 06:20:01.761135 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>2015-05-18 06:20:01.761138 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>2015-05-18 06:20:01.761141 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>2015-05-18 06:20:01.761445 7f09890a87c0  1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1
>2015-05-18 06:20:01.761932 7f09890a87c0  0 mon.Mon@-1(probing) e1  my rank is now 0 (was -1)
>2015-05-18 06:20:01.761953 7f09890a87c0  1 mon.Mon@0(probing) e1 win_standalone_election
>2015-05-18 06:20:01.801222 7f09890a87c0  0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0
>2015-05-18 06:20:01.801309 7f09890a87c0  0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
>2015-05-18 06:20:01.801372 7f09890a87c0  0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>2015-05-18 06:20:01.801422 7f09890a87c0  0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up
>2015-05-18 06:20:01.801492 7f09890a87c0  0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in
>2015-05-18 06:20:02.475336 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:02.475350 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:02.475405 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:02.475418 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:02.823300 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
>2015-05-18 06:20:02.823374 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
>2015-05-18 06:20:02.832689 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:02.832704 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:02.832760 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:02.832773 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:07.802963 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
>2015-05-18 06:20:07.803031 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
>2015-05-18 06:20:07.835601 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:07.835627 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:07.835680 7f09817a0700  1 mon.Mon@0(leader).osd e84  we have enough reports/reporters to mark osd.0 down
>2015-05-18 06:20:07.835700 7f09817a0700  0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
>2015-05-18 06:20:07.835868 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>2015-05-18 06:20:07.835888 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>2015-05-18 06:20:07.835925 7f09817a0700  1 mon.Mon@0(leader).osd e84  we have enough reports/reporters to mark osd.3 down
>2015-05-18 06:20:07.835939 7f09817a0700  0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
>2015-05-18 06:20:07.937432 7f098370b700  1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in
>2015-05-18 06:20:07.987480 7f098370b700  0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in
>2015-05-18 06:20:08.013617 7f098370b700  0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>2015-05-18 06:20:08.949801 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1
>2015-05-18 06:20:08.949836 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
>2015-05-18 06:20:08.951357 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
>2015-05-18 06:20:08.951380 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>2015-05-18 06:20:08.952829 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1
>2015-05-18 06:20:08.952852 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
>2015-05-18 06:20:08.954227 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1
>2015-05-18 06:20:08.954251 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
>2015-05-18 06:20:09.143603 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
>2015-05-18 06:20:09.143635 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>2015-05-18 06:20:09.145494 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1
>2015-05-18 06:20:09.145520 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
>2015-05-18 06:20:09.147309 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1
>2015-05-18 06:20:09.147337 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
>2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) **
> in thread 7f09817a0700
>
> ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
> 1: /usr/bin/ceph-mon() [0x9017e2]
> 2: (()+0xf130) [0x7f098826e130]
> 3: (gsignal()+0x37) [0x7f0986c885d7]
> 4: (abort()+0x148) [0x7f0986c89cc8]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5]
> 6: (()+0x5e926) [0x7f098758a926]
> 7: (()+0x5e953) [0x7f098758a953]
> 8: (()+0x5eb73) [0x7f098758ab73]
> 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717]
> 10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561]
> 11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918]
> 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9]
> 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9]
> 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5]
> 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b]
> 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3]
> 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c]
> 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89]
> 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936]
> 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403]
> 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a]
> 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d]
> 23: (()+0x7df5) [0x7f0988266df5]
> 24: (clone()+0x6d) [0x7f0986d491ad]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>--- begin dump of recent events ---
>  -448> 2015-05-18 06:20:01.642177 7f09890a87c0  5 asok(0x4650000) register_command perfcounters_dump hook 0x45c0050
>  -447> 2015-05-18 06:20:01.642205 7f09890a87c0  5 asok(0x4650000) register_command 1 hook 0x45c0050
>  -446> 2015-05-18 06:20:01.642212 7f09890a87c0  5 asok(0x4650000) register_command perf dump hook 0x45c0050
>  -445> 2015-05-18 06:20:01.642219 7f09890a87c0  5 asok(0x4650000) register_command perfcounters_schema hook 0x45c0050
>  -444> 2015-05-18 06:20:01.642224 7f09890a87c0  5 asok(0x4650000) register_command 2 hook 0x45c0050
>  -443> 2015-05-18 06:20:01.642228 7f09890a87c0  5 asok(0x4650000) register_command perf schema hook 0x45c0050
>  -442> 2015-05-18 06:20:01.642233 7f09890a87c0  5 asok(0x4650000) register_command perf reset hook 0x45c0050
>  -441> 2015-05-18 06:20:01.642237 7f09890a87c0  5 asok(0x4650000) register_command config show hook 0x45c0050
>  -440> 2015-05-18 06:20:01.642242 7f09890a87c0  5 asok(0x4650000) register_command config set hook 0x45c0050
>  -439> 2015-05-18 06:20:01.642247 7f09890a87c0  5 asok(0x4650000) register_command config get hook 0x45c0050
>  -438> 2015-05-18 06:20:01.642251 7f09890a87c0  5 asok(0x4650000) register_command config diff hook 0x45c0050
>  -437> 2015-05-18 06:20:01.642255 7f09890a87c0  5 asok(0x4650000) register_command log flush hook 0x45c0050
>  -436> 2015-05-18 06:20:01.642260 7f09890a87c0  5 asok(0x4650000) register_command log dump hook 0x45c0050
>  -435> 2015-05-18 06:20:01.642264 7f09890a87c0  5 asok(0x4650000) register_command log reopen hook 0x45c0050
>  -434> 2015-05-18 06:20:01.644938 7f09890a87c0  0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992
>  -433> 2015-05-18 06:20:01.667692 7f09890a87c0  5 asok(0x4650000) init /var/run/ceph/ceph-mon.Mon.asok
>  -432> 2015-05-18 06:20:01.667706 7f09890a87c0  5 asok(0x4650000) bind_and_listen /var/run/ceph/ceph-mon.Mon.asok
>  -431> 2015-05-18 06:20:01.667808 7f09890a87c0  5 asok(0x4650000) register_command 0 hook 0x45b80b8
>  -430> 2015-05-18 06:20:01.667817 7f09890a87c0  5 asok(0x4650000) register_command version hook 0x45b80b8
>  -429> 2015-05-18 06:20:01.667823 7f09890a87c0  5 asok(0x4650000) register_command git_version hook 0x45b80b8
>  -428> 2015-05-18 06:20:01.667828 7f09890a87c0  5 asok(0x4650000) register_command help hook 0x45c00b0
>  -427> 2015-05-18 06:20:01.667833 7f09890a87c0  5 asok(0x4650000) register_command get_command_descriptions hook 0x45c0150
>  -426> 2015-05-18 06:20:01.667864 7f0984a8b700  5 asok(0x4650000) entry start
>  -425> 2015-05-18 06:20:01.742013 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 1496 bytes
>  -424> 2015-05-18 06:20:01.759779 7f09890a87c0  0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
>  -423> 2015-05-18 06:20:01.759829 7f09890a87c0  1 -- 10.8.128.33:6789/0 learned my addr 10.8.128.33:6789/0
>  -422> 2015-05-18 06:20:01.759835 7f09890a87c0  1 accepter.accepter.bind my_inst.addr is 10.8.128.33:6789/0 need_addr=0
>  -421> 2015-05-18 06:20:01.759884 7f09890a87c0  5 adding auth protocol: cephx
>  -420> 2015-05-18 06:20:01.759890 7f09890a87c0  5 adding auth protocol: cephx
>  -419> 2015-05-18 06:20:01.759909 7f09890a87c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
>  -418> 2015-05-18 06:20:01.759913 7f09890a87c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info)
>  -417> 2015-05-18 06:20:01.759988 7f09890a87c0  1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36
>  -416> 2015-05-18 06:20:01.760303 7f09890a87c0  1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1
>  -415> 2015-05-18 06:20:01.760314 7f09890a87c0  1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap
>  -414> 2015-05-18 06:20:01.760919 7f09890a87c0  4 mon.Mon@-1(probing).mds e1 new map
>  -413> 2015-05-18 06:20:01.760927 7f09890a87c0  0 mon.Mon@-1(probing).mds e1 print_map
>epoch	1
>flags	0
>created	0.000000
>modified	2015-05-06 15:39:42.874456
>tableserver	0
>root	0
>session_timeout	0
>session_autoclose	0
>max_file_size	0
>last_failure	0
>last_failure_osd_epoch	0
>compat	compat={},rocompat={},incompat={}
>max_mds	0
>in	
>up	{}
>failed	
>stopped	
>data_pools	
>metadata_pool	0
>inline_data	disabled
>
>  -412> 2015-05-18 06:20:01.761127 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>  -411> 2015-05-18 06:20:01.761135 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>  -410> 2015-05-18 06:20:01.761138 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>  -409> 2015-05-18 06:20:01.761141 7f09890a87c0  0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires
>  -408> 2015-05-18 06:20:01.761445 7f09890a87c0  1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1
>  -407> 2015-05-18 06:20:01.761792 7f09890a87c0  2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-Mon/keyring
>  -406> 2015-05-18 06:20:01.761799 7f09890a87c0  5 asok(0x4650000) register_command mon_status hook 0x45c0330
>  -405> 2015-05-18 06:20:01.761804 7f09890a87c0  5 asok(0x4650000) register_command quorum_status hook 0x45c0330
>  -404> 2015-05-18 06:20:01.761808 7f09890a87c0  5 asok(0x4650000) register_command sync_force hook 0x45c0330
>  -403> 2015-05-18 06:20:01.761811 7f09890a87c0  5 asok(0x4650000) register_command add_bootstrap_peer_hint hook 0x45c0330
>  -402> 2015-05-18 06:20:01.761814 7f09890a87c0  5 asok(0x4650000) register_command quorum enter hook 0x45c0330
>  -401> 2015-05-18 06:20:01.761817 7f09890a87c0  5 asok(0x4650000) register_command quorum exit hook 0x45c0330
>  -400> 2015-05-18 06:20:01.761825 7f09890a87c0  1 -- 10.8.128.33:6789/0 messenger.start
>  -399> 2015-05-18 06:20:01.761854 7f09890a87c0  2 mon.Mon@-1(probing) e1 init
>  -398> 2015-05-18 06:20:01.761912 7f09890a87c0  1 accepter.accepter.start
>  -397> 2015-05-18 06:20:01.761932 7f09890a87c0  0 mon.Mon@-1(probing) e1  my rank is now 0 (was -1)
>  -396> 2015-05-18 06:20:01.761939 7f09890a87c0  1 -- 10.8.128.33:6789/0 mark_down_all
>  -395> 2015-05-18 06:20:01.761953 7f09890a87c0  1 mon.Mon@0(probing) e1 win_standalone_election
>  -394> 2015-05-18 06:20:01.801222 7f09890a87c0  0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0
>  -393> 2015-05-18 06:20:01.801230 7f09890a87c0 10 log_client _send_to_monlog to self
>  -392> 2015-05-18 06:20:01.801231 7f09890a87c0 10 log_client  log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
>  -391> 2015-05-18 06:20:01.801235 7f09890a87c0 10 log_client  will send 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0
>  -390> 2015-05-18 06:20:01.801247 7f09890a87c0  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
>  -389> 2015-05-18 06:20:01.801309 7f09890a87c0  0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
>  -388> 2015-05-18 06:20:01.801316 7f09890a87c0 10 log_client _send_to_monlog to self
>  -387> 2015-05-18 06:20:01.801317 7f09890a87c0 10 log_client  log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
>  -386> 2015-05-18 06:20:01.801298 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
>  -385> 2015-05-18 06:20:01.801320 7f09890a87c0 10 log_client  will send 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
>  -384> 2015-05-18 06:20:01.801327 7f09890a87c0  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600
>  -383> 2015-05-18 06:20:01.801372 7f09890a87c0  0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -382> 2015-05-18 06:20:01.801376 7f09890a87c0 10 log_client _send_to_monlog to self
>  -381> 2015-05-18 06:20:01.801377 7f09890a87c0 10 log_client  log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
>  -380> 2015-05-18 06:20:01.801380 7f09890a87c0 10 log_client  will send 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -379> 2015-05-18 06:20:01.801387 7f09890a87c0  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2880 con 0x4609600
>  -378> 2015-05-18 06:20:01.801422 7f09890a87c0  0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up
>  -377> 2015-05-18 06:20:01.801426 7f09890a87c0 10 log_client _send_to_monlog to self
>  -376> 2015-05-18 06:20:01.801427 7f09890a87c0 10 log_client  log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
>  -375> 2015-05-18 06:20:01.801428 7f09890a87c0 10 log_client  will send 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up
>  -374> 2015-05-18 06:20:01.801435 7f09890a87c0  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2640 con 0x4609600
>  -373> 2015-05-18 06:20:01.801492 7f09890a87c0  0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in
>  -372> 2015-05-18 06:20:01.801496 7f09890a87c0 10 log_client _send_to_monlog to self
>  -371> 2015-05-18 06:20:01.801498 7f09890a87c0 10 log_client  log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
>  -370> 2015-05-18 06:20:01.801499 7f09890a87c0 10 log_client  will send 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in
>  -369> 2015-05-18 06:20:01.801506 7f09890a87c0  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
>  -368> 2015-05-18 06:20:01.801773 7f09817a0700  5 mon.Mon@0(leader) e1 setting monitor caps on this connection
>  -367> 2015-05-18 06:20:01.801790 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801791 lease_expire=0.000000 has v0 lc 78623
>  -366> 2015-05-18 06:20:01.801820 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600
>  -365> 2015-05-18 06:20:01.801832 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801833 lease_expire=0.000000 has v0 lc 78623
>  -364> 2015-05-18 06:20:01.801846 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2880 con 0x4609600
>  -363> 2015-05-18 06:20:01.801853 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801854 lease_expire=0.000000 has v0 lc 78623
>  -362> 2015-05-18 06:20:01.801866 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2640 con 0x4609600
>  -361> 2015-05-18 06:20:01.801874 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801874 lease_expire=0.000000 has v0 lc 78623
>  -360> 2015-05-18 06:20:01.801886 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
>  -359> 2015-05-18 06:20:01.801894 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801894 lease_expire=0.000000 has v0 lc 78623
>  -358> 2015-05-18 06:20:01.851959 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) queue_pending_finisher 0x45c04d0
>  -357> 2015-05-18 06:20:01.918852 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 1) v1 -- ?+0 0x47b81e0 con 0x4609600
>  -356> 2015-05-18 06:20:01.918871 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 2) v1 -- ?+0 0x47b85a0 con 0x4609600
>  -355> 2015-05-18 06:20:01.918879 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 3) v1 -- ?+0 0x47b83c0 con 0x4609600
>  -354> 2015-05-18 06:20:01.918886 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 4) v1 -- ?+0 0x47b8b40 con 0x4609600
>  -353> 2015-05-18 06:20:01.918894 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 5) v1 -- ?+0 0x47b8960 con 0x4609600
>  -352> 2015-05-18 06:20:01.918930 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
>  -351> 2015-05-18 06:20:01.918953 7f09817a0700 10 log_client handle_log_ack log(last 1) v1
>  -350> 2015-05-18 06:20:01.918956 7f09817a0700 10 log_client  logged 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0
>  -349> 2015-05-18 06:20:01.918977 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600
>  -348> 2015-05-18 06:20:01.918986 7f09817a0700 10 log_client handle_log_ack log(last 2) v1
>  -347> 2015-05-18 06:20:01.918989 7f09817a0700 10 log_client  logged 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0}
>  -346> 2015-05-18 06:20:01.918997 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
>  -345> 2015-05-18 06:20:01.919005 7f09817a0700 10 log_client handle_log_ack log(last 3) v1
>  -344> 2015-05-18 06:20:01.919007 7f09817a0700 10 log_client  logged 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -343> 2015-05-18 06:20:01.919015 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
>  -342> 2015-05-18 06:20:01.919022 7f09817a0700 10 log_client handle_log_ack log(last 4) v1
>  -341> 2015-05-18 06:20:01.919025 7f09817a0700 10 log_client  logged 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up
>  -340> 2015-05-18 06:20:01.919034 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
>  -339> 2015-05-18 06:20:01.919041 7f09817a0700 10 log_client handle_log_ack log(last 5) v1
>  -338> 2015-05-18 06:20:01.919043 7f09817a0700 10 log_client  logged 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in
>  -337> 2015-05-18 06:20:02.473956 7f0989044700  1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48d0000 sd=20 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a260).accept sd=20 10.8.128.86:48717/0
>  -336> 2015-05-18 06:20:02.474299 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 1 ==== auth(proto 0 26 bytes epoch 1) v1 ==== 56+0+0 (1708591557 0 0) 0x47f3a80 con 0x460a260
>  -335> 2015-05-18 06:20:02.474322 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474323 lease_expire=0.000000 has v0 lc 78624
>  -334> 2015-05-18 06:20:02.474362 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2f40 con 0x460a260
>  -333> 2015-05-18 06:20:02.474629 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1575692119 0 0) 0x47f4140 con 0x460a260
>  -332> 2015-05-18 06:20:02.474645 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474646 lease_expire=0.000000 has v0 lc 78624
>  -331> 2015-05-18 06:20:02.474885 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f3a80 con 0x460a260
>  -330> 2015-05-18 06:20:02.475200 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 3 ==== mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 ==== 50+0+0 (3889722309 0 0) 0x466a600 con 0x460a260
>  -329> 2015-05-18 06:20:02.475267 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a260
>  -328> 2015-05-18 06:20:02.475300 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 4 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 765sec e84 v84) v3 ==== 188+0+0 (2601468610 0 0) 0x46798c0 con 0x460a260
>  -327> 2015-05-18 06:20:02.475316 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475317 lease_expire=0.000000 has v0 lc 78624
>  -326> 2015-05-18 06:20:02.475336 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -325> 2015-05-18 06:20:02.475350 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -324> 2015-05-18 06:20:02.475353 7f09817a0700 10 log_client _send_to_monlog to self
>  -323> 2015-05-18 06:20:02.475354 7f09817a0700 10 log_client  log_queue is 1 last_log 6 sent 5 num 1 unsent 1 sending 1
>  -322> 2015-05-18 06:20:02.475356 7f09817a0700 10 log_client  will send 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -321> 2015-05-18 06:20:02.475362 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600
>  -320> 2015-05-18 06:20:02.475385 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 5 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x4679e40 con 0x460a260
>  -319> 2015-05-18 06:20:02.475396 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475397 lease_expire=0.000000 has v0 lc 78624
>  -318> 2015-05-18 06:20:02.475405 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -317> 2015-05-18 06:20:02.475418 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -316> 2015-05-18 06:20:02.475420 7f09817a0700 10 log_client _send_to_monlog to self
>  -315> 2015-05-18 06:20:02.475421 7f09817a0700 10 log_client  log_queue is 2 last_log 7 sent 6 num 2 unsent 1 sending 1
>  -314> 2015-05-18 06:20:02.475423 7f09817a0700 10 log_client  will send 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -313> 2015-05-18 06:20:02.475428 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600
>  -312> 2015-05-18 06:20:02.475444 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f3cc0 con 0x4609600
>  -311> 2015-05-18 06:20:02.475452 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475453 lease_expire=0.000000 has v0 lc 78624
>  -310> 2015-05-18 06:20:02.475470 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600
>  -309> 2015-05-18 06:20:02.475479 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475480 lease_expire=0.000000 has v0 lc 78624
>  -308> 2015-05-18 06:20:02.475496 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 6 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=84}) v2 ==== 69+0+0 (2997052209 0 0) 0x466b000 con 0x460a260
>  -307> 2015-05-18 06:20:02.475517 7f09817a0700  5 mon.Mon@0(leader).osd e84 send_incremental [84..84] to osd.1 10.8.128.86:6800/19549
>  -306> 2015-05-18 06:20:02.475587 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..84 src has 1..84) v3 -- ?+0 0x47f4140 con 0x460a260
>  -305> 2015-05-18 06:20:02.475605 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b83c0 con 0x460a260
>  -304> 2015-05-18 06:20:02.815632 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
>  -303> 2015-05-18 06:20:02.823300 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
>  -302> 2015-05-18 06:20:02.823308 7f0984a8b700 10 log_client _send_to_monlog to self
>  -301> 2015-05-18 06:20:02.823309 7f0984a8b700 10 log_client  log_queue is 3 last_log 8 sent 7 num 3 unsent 1 sending 1
>  -300> 2015-05-18 06:20:02.823312 7f0984a8b700 10 log_client  will send 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
>  -299> 2015-05-18 06:20:02.823333 7f0984a8b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648d80 con 0x4609600
>  -298> 2015-05-18 06:20:02.823374 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
>  -297> 2015-05-18 06:20:02.823377 7f0984a8b700 10 log_client _send_to_monlog to self
>  -296> 2015-05-18 06:20:02.823378 7f0984a8b700 10 log_client  log_queue is 4 last_log 9 sent 8 num 4 unsent 1 sending 1
>  -295> 2015-05-18 06:20:02.823380 7f0984a8b700 10 log_client  will send 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
>  -294> 2015-05-18 06:20:02.823386 7f0984a8b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600
>  -293> 2015-05-18 06:20:02.823393 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648d80 con 0x4609600
>  -292> 2015-05-18 06:20:02.823407 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 520 bytes
>  -291> 2015-05-18 06:20:02.823421 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823422 lease_expire=0.000000 has v0 lc 78624
>  -290> 2015-05-18 06:20:02.823450 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600
>  -289> 2015-05-18 06:20:02.823467 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823468 lease_expire=0.000000 has v0 lc 78624
>  -288> 2015-05-18 06:20:02.832644 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 7 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 766sec e84 v84) v3 ==== 188+0+0 (4180528123 0 0) 0x4679b80 con 0x460a260
>  -287> 2015-05-18 06:20:02.832671 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832672 lease_expire=0.000000 has v0 lc 78624
>  -286> 2015-05-18 06:20:02.832689 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -285> 2015-05-18 06:20:02.832704 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -284> 2015-05-18 06:20:02.832707 7f09817a0700 10 log_client _send_to_monlog to self
>  -283> 2015-05-18 06:20:02.832708 7f09817a0700 10 log_client  log_queue is 5 last_log 10 sent 9 num 5 unsent 1 sending 1
>  -282> 2015-05-18 06:20:02.832710 7f09817a0700 10 log_client  will send 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -281> 2015-05-18 06:20:02.832717 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
>  -280> 2015-05-18 06:20:02.832739 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 8 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x467a680 con 0x460a260
>  -279> 2015-05-18 06:20:02.832751 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832751 lease_expire=0.000000 has v0 lc 78624
>  -278> 2015-05-18 06:20:02.832760 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -277> 2015-05-18 06:20:02.832773 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -276> 2015-05-18 06:20:02.832775 7f09817a0700 10 log_client _send_to_monlog to self
>  -275> 2015-05-18 06:20:02.832777 7f09817a0700 10 log_client  log_queue is 6 last_log 11 sent 10 num 6 unsent 1 sending 1
>  -274> 2015-05-18 06:20:02.832779 7f09817a0700 10 log_client  will send 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -273> 2015-05-18 06:20:02.832785 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600
>  -272> 2015-05-18 06:20:02.832807 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
>  -271> 2015-05-18 06:20:02.832817 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832817 lease_expire=0.000000 has v0 lc 78624
>  -270> 2015-05-18 06:20:02.832833 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600
>  -269> 2015-05-18 06:20:02.832842 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832844 lease_expire=0.000000 has v0 lc 78624
>  -268> 2015-05-18 06:20:02.918393 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) queue_pending_finisher 0x45c04c0
>  -267> 2015-05-18 06:20:02.977214 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 6) v1 -- ?+0 0x47b8780 con 0x4609600
>  -266> 2015-05-18 06:20:02.977246 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 7) v1 -- ?+0 0x47b81e0 con 0x4609600
>  -265> 2015-05-18 06:20:02.977254 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 8) v1 -- ?+0 0x47b8b40 con 0x4609600
>  -264> 2015-05-18 06:20:02.977259 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 9) v1 -- ?+0 0x47b83c0 con 0x4609600
>  -263> 2015-05-18 06:20:02.977267 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 10) v1 -- ?+0 0x47b8960 con 0x4609600
>  -262> 2015-05-18 06:20:02.977291 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 11) v1 -- ?+0 0x47b92c0 con 0x4609600
>  -261> 2015-05-18 06:20:02.977297 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 6) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600
>  -260> 2015-05-18 06:20:02.977319 7f09817a0700 10 log_client handle_log_ack log(last 6) v1
>  -259> 2015-05-18 06:20:02.977322 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -258> 2015-05-18 06:20:02.977334 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
>  -257> 2015-05-18 06:20:02.977341 7f09817a0700 10 log_client handle_log_ack log(last 7) v1
>  -256> 2015-05-18 06:20:02.977343 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -255> 2015-05-18 06:20:02.977352 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 8) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
>  -254> 2015-05-18 06:20:02.977375 7f09817a0700 10 log_client handle_log_ack log(last 8) v1
>  -253> 2015-05-18 06:20:02.977379 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
>  -252> 2015-05-18 06:20:02.977388 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 9) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
>  -251> 2015-05-18 06:20:02.977395 7f09817a0700 10 log_client handle_log_ack log(last 9) v1
>  -250> 2015-05-18 06:20:02.977398 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
>  -249> 2015-05-18 06:20:02.977405 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 10) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
>  -248> 2015-05-18 06:20:02.977412 7f09817a0700 10 log_client handle_log_ack log(last 10) v1
>  -247> 2015-05-18 06:20:02.977414 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -246> 2015-05-18 06:20:02.977422 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 11) v1 ==== 0+0+0 (0 0 0) 0x47b92c0 con 0x4609600
>  -245> 2015-05-18 06:20:02.977430 7f09817a0700 10 log_client handle_log_ack log(last 11) v1
>  -244> 2015-05-18 06:20:02.977431 7f09817a0700 10 log_client  logged 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -243> 2015-05-18 06:20:06.761957 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761960 lease_expire=0.000000 has v0 lc 78625
>  -242> 2015-05-18 06:20:06.761973 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761974 lease_expire=0.000000 has v0 lc 78625
>  -241> 2015-05-18 06:20:06.761993 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761993 lease_expire=0.000000 has v0 lc 78625
>  -240> 2015-05-18 06:20:06.762078 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) queue_pending_finisher 0x45c04f0
>  -239> 2015-05-18 06:20:07.793435 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
>  -238> 2015-05-18 06:20:07.802963 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
>  -237> 2015-05-18 06:20:07.802972 7f0984a8b700 10 log_client _send_to_monlog to self
>  -236> 2015-05-18 06:20:07.802973 7f0984a8b700 10 log_client  log_queue is 1 last_log 12 sent 11 num 1 unsent 1 sending 1
>  -235> 2015-05-18 06:20:07.802977 7f0984a8b700 10 log_client  will send 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
>  -234> 2015-05-18 06:20:07.802995 7f0984a8b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648900 con 0x4609600
>  -233> 2015-05-18 06:20:07.803031 7f0984a8b700  0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
>  -232> 2015-05-18 06:20:07.803033 7f0984a8b700 10 log_client _send_to_monlog to self
>  -231> 2015-05-18 06:20:07.803034 7f0984a8b700 10 log_client  log_queue is 2 last_log 13 sent 12 num 2 unsent 1 sending 1
>  -230> 2015-05-18 06:20:07.803037 7f0984a8b700 10 log_client  will send 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
>  -229> 2015-05-18 06:20:07.803043 7f0984a8b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
>  -228> 2015-05-18 06:20:07.803059 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 307 bytes
>  -227> 2015-05-18 06:20:07.803053 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648900 con 0x4609600
>  -226> 2015-05-18 06:20:07.803083 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803084 lease_expire=0.000000 has v0 lc 78626
>  -225> 2015-05-18 06:20:07.803114 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
>  -224> 2015-05-18 06:20:07.803123 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803125 lease_expire=0.000000 has v0 lc 78626
>  -223> 2015-05-18 06:20:07.803590 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
>  -222> 2015-05-18 06:20:07.810261 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) queue_pending_finisher 0x45c0320
>  -221> 2015-05-18 06:20:07.812704 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'version' '' to 0x45b80b8 returned 20 bytes
>  -220> 2015-05-18 06:20:07.833354 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 9 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 771sec e84 v84) v3 ==== 188+0+0 (3406336582 0 0) 0x467a3c0 con 0x460a260
>  -219> 2015-05-18 06:20:07.835570 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835574 lease_expire=0.000000 has v0 lc 78626
>  -218> 2015-05-18 06:20:07.835601 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -217> 2015-05-18 06:20:07.835627 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -216> 2015-05-18 06:20:07.835632 7f09817a0700 10 log_client _send_to_monlog to self
>  -215> 2015-05-18 06:20:07.835633 7f09817a0700 10 log_client  log_queue is 3 last_log 14 sent 13 num 3 unsent 1 sending 1
>  -214> 2015-05-18 06:20:07.835639 7f09817a0700 10 log_client  will send 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -213> 2015-05-18 06:20:07.835652 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4ec0 con 0x4609600
>  -212> 2015-05-18 06:20:07.835680 7f09817a0700  1 mon.Mon@0(leader).osd e84  we have enough reports/reporters to mark osd.0 down
>  -211> 2015-05-18 06:20:07.835700 7f09817a0700  0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
>  -210> 2015-05-18 06:20:07.835703 7f09817a0700 10 log_client _send_to_monlog to self
>  -209> 2015-05-18 06:20:07.835704 7f09817a0700 10 log_client  log_queue is 4 last_log 15 sent 14 num 4 unsent 1 sending 1
>  -208> 2015-05-18 06:20:07.835707 7f09817a0700 10 log_client  will send 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
>  -207> 2015-05-18 06:20:07.835722 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4c80 con 0x4609600
>  -206> 2015-05-18 06:20:07.835744 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4ec0 con 0x4609600
>  -205> 2015-05-18 06:20:07.835759 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835761 lease_expire=0.000000 has v0 lc 78626
>  -204> 2015-05-18 06:20:07.835780 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4c80 con 0x4609600
>  -203> 2015-05-18 06:20:07.835809 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835810 lease_expire=0.000000 has v0 lc 78626
>  -202> 2015-05-18 06:20:07.835826 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 10 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 728sec e84 v84) v3 ==== 188+0+0 (2916513576 0 0) 0x467a100 con 0x460a260
>  -201> 2015-05-18 06:20:07.835853 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835854 lease_expire=0.000000 has v0 lc 78626
>  -200> 2015-05-18 06:20:07.835868 7f09817a0700  1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1
>  -199> 2015-05-18 06:20:07.835888 7f09817a0700  0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -198> 2015-05-18 06:20:07.835892 7f09817a0700 10 log_client _send_to_monlog to self
>  -197> 2015-05-18 06:20:07.835893 7f09817a0700 10 log_client  log_queue is 5 last_log 16 sent 15 num 5 unsent 1 sending 1
>  -196> 2015-05-18 06:20:07.835897 7f09817a0700 10 log_client  will send 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -195> 2015-05-18 06:20:07.835907 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4a40 con 0x4609600
>  -194> 2015-05-18 06:20:07.835925 7f09817a0700  1 mon.Mon@0(leader).osd e84  we have enough reports/reporters to mark osd.3 down
>  -193> 2015-05-18 06:20:07.835939 7f09817a0700  0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
>  -192> 2015-05-18 06:20:07.835942 7f09817a0700 10 log_client _send_to_monlog to self
>  -191> 2015-05-18 06:20:07.835943 7f09817a0700 10 log_client  log_queue is 6 last_log 17 sent 16 num 6 unsent 1 sending 1
>  -190> 2015-05-18 06:20:07.835946 7f09817a0700 10 log_client  will send 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
>  -189> 2015-05-18 06:20:07.835956 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5c40 con 0x4609600
>  -188> 2015-05-18 06:20:07.835984 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4a40 con 0x4609600
>  -187> 2015-05-18 06:20:07.835997 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835998 lease_expire=0.000000 has v0 lc 78626
>  -186> 2015-05-18 06:20:07.836011 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5c40 con 0x4609600
>  -185> 2015-05-18 06:20:07.836021 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.836022 lease_expire=0.000000 has v0 lc 78626
>  -184> 2015-05-18 06:20:07.860966 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 12) v1 -- ?+0 0x47b90e0 con 0x4609600
>  -183> 2015-05-18 06:20:07.860985 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 13) v1 -- ?+0 0x47b8f00 con 0x4609600
>  -182> 2015-05-18 06:20:07.860993 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.860994 lease_expire=0.000000 has v0 lc 78627
>  -181> 2015-05-18 06:20:07.861010 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861011 lease_expire=0.000000 has v0 lc 78627
>  -180> 2015-05-18 06:20:07.861023 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861024 lease_expire=0.000000 has v0 lc 78627
>  -179> 2015-05-18 06:20:07.861034 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861035 lease_expire=0.000000 has v0 lc 78627
>  -178> 2015-05-18 06:20:07.861029 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 12) v1 ==== 0+0+0 (0 0 0) 0x47b90e0 con 0x4609600
>  -177> 2015-05-18 06:20:07.861052 7f09817a0700 10 log_client handle_log_ack log(last 12) v1
>  -176> 2015-05-18 06:20:07.861057 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
>  -175> 2015-05-18 06:20:07.861074 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 13) v1 ==== 0+0+0 (0 0 0) 0x47b8f00 con 0x4609600
>  -174> 2015-05-18 06:20:07.861086 7f09817a0700 10 log_client handle_log_ack log(last 13) v1
>  -173> 2015-05-18 06:20:07.861090 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
>  -172> 2015-05-18 06:20:07.885804 7f0981fa1700  2 mon.Mon@0(leader).osd e84  osd.0 DOWN
>  -171> 2015-05-18 06:20:07.885814 7f0981fa1700  2 mon.Mon@0(leader).osd e84  osd.3 DOWN
>  -170> 2015-05-18 06:20:07.885893 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) queue_pending_finisher 0x45c04d0
>  -169> 2015-05-18 06:20:07.937432 7f098370b700  1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in
>  -168> 2015-05-18 06:20:07.962214 7f098370b700  5 mon.Mon@0(leader).paxos(paxos refresh c 78062..78628) is_readable = 0 - now=2015-05-18 06:20:07.962218 lease_expire=0.000000 has v0 lc 78628
>  -167> 2015-05-18 06:20:07.962309 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..85 src has 1..85) v3 -- ?+0 0x47f2400 con 0x460a260
>  -166> 2015-05-18 06:20:07.962334 7f098370b700  5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84
>  -165> 2015-05-18 06:20:07.962341 7f098370b700  5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549
>  -164> 2015-05-18 06:20:07.962379 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2f40 con 0x460a260
>  -163> 2015-05-18 06:20:07.962394 7f098370b700  5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84
>  -162> 2015-05-18 06:20:07.962399 7f098370b700  5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549
>  -161> 2015-05-18 06:20:07.962590 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.962592 lease_expire=0.000000 has v0 lc 78628
>  -160> 2015-05-18 06:20:07.962922 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) queue_pending_finisher 0x45c04d0
>  -159> 2015-05-18 06:20:07.987480 7f098370b700  0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in
>  -158> 2015-05-18 06:20:07.987489 7f098370b700 10 log_client _send_to_monlog to self
>  -157> 2015-05-18 06:20:07.987492 7f098370b700 10 log_client  log_queue is 5 last_log 18 sent 17 num 5 unsent 1 sending 1
>  -156> 2015-05-18 06:20:07.987495 7f098370b700 10 log_client  will send 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in
>  -155> 2015-05-18 06:20:07.987506 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600
>  -154> 2015-05-18 06:20:07.987564 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600
>  -153> 2015-05-18 06:20:07.987582 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.987583 lease_expire=0.000000 has v0 lc 78628
>  -152> 2015-05-18 06:20:08.005319 7f09817a0700  1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 11 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=86}) v2 ==== 69+0+0 (1597159760 0 0) 0x466b400 con 0x460a260
>  -151> 2015-05-18 06:20:08.005352 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8f00 con 0x460a260
>  -150> 2015-05-18 06:20:08.013617 7f098370b700  0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -149> 2015-05-18 06:20:08.013624 7f098370b700 10 log_client _send_to_monlog to self
>  -148> 2015-05-18 06:20:08.013625 7f098370b700 10 log_client  log_queue is 6 last_log 19 sent 18 num 6 unsent 1 sending 1
>  -147> 2015-05-18 06:20:08.013627 7f098370b700 10 log_client  will send 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -146> 2015-05-18 06:20:08.013637 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600
>  -145> 2015-05-18 06:20:08.013689 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600
>  -144> 2015-05-18 06:20:08.013702 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.013703 lease_expire=0.000000 has v0 lc 78629
>  -143> 2015-05-18 06:20:08.860774 7f0981fa1700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) queue_pending_finisher 0x45c0510
>  -142> 2015-05-18 06:20:08.909048 7f097f69b700  1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a3c0).accept sd=21 10.8.128.33:60116/0
>  -141> 2015-05-18 06:20:08.909248 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f4140 con 0x460a3c0
>  -140> 2015-05-18 06:20:08.909277 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.909278 lease_expire=0.000000 has v0 lc 78629
>  -139> 2015-05-18 06:20:08.909363 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) queue_pending_finisher 0x45c0500
>  -138> 2015-05-18 06:20:08.913369 7f098370b700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.913371 lease_expire=0.000000 has v0 lc 78630
>  -137> 2015-05-18 06:20:08.913393 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x47f2ac0 con 0x460a3c0
>  -136> 2015-05-18 06:20:08.913421 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4648900 con 0x460a3c0
>  -135> 2015-05-18 06:20:08.913486 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 14) v1 -- ?+0 0x47b85a0 con 0x4609600
>  -134> 2015-05-18 06:20:08.913503 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 15) v1 -- ?+0 0x47b8960 con 0x4609600
>  -133> 2015-05-18 06:20:08.913512 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 16) v1 -- ?+0 0x47b83c0 con 0x4609600
>  -132> 2015-05-18 06:20:08.913519 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 17) v1 -- ?+0 0x47b8b40 con 0x4609600
>  -131> 2015-05-18 06:20:08.913528 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 18) v1 -- ?+0 0x47b81e0 con 0x4609600
>  -130> 2015-05-18 06:20:08.913536 7f098370b700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 19) v1 -- ?+0 0x47b8780 con 0x4609600
>  -129> 2015-05-18 06:20:08.913547 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 14) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600
>  -128> 2015-05-18 06:20:08.938024 7f09817a0700 10 log_client handle_log_ack log(last 14) v1
>  -127> 2015-05-18 06:20:08.938034 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549
>  -126> 2015-05-18 06:20:08.938051 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 15) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600
>  -125> 2015-05-18 06:20:08.938060 7f09817a0700 10 log_client handle_log_ack log(last 15) v1
>  -124> 2015-05-18 06:20:08.938063 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211)
>  -123> 2015-05-18 06:20:08.938071 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 16) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600
>  -122> 2015-05-18 06:20:08.938079 7f09817a0700 10 log_client handle_log_ack log(last 16) v1
>  -121> 2015-05-18 06:20:08.938083 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549
>  -120> 2015-05-18 06:20:08.938092 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 17) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600
>  -119> 2015-05-18 06:20:08.938099 7f09817a0700 10 log_client handle_log_ack log(last 17) v1
>  -118> 2015-05-18 06:20:08.938101 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015)
>  -117> 2015-05-18 06:20:08.938109 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 18) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600
>  -116> 2015-05-18 06:20:08.938116 7f09817a0700 10 log_client handle_log_ack log(last 18) v1
>  -115> 2015-05-18 06:20:08.938119 7f09817a0700 10 log_client  logged 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in
>  -114> 2015-05-18 06:20:08.938127 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 19) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600
>  -113> 2015-05-18 06:20:08.938135 7f09817a0700 10 log_client handle_log_ack log(last 19) v1
>  -112> 2015-05-18 06:20:08.938137 7f09817a0700 10 log_client  logged 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail
>  -111> 2015-05-18 06:20:08.938145 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (309592415 0 0) 0x47f2ac0 con 0x460a3c0
>  -110> 2015-05-18 06:20:08.938154 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938155 lease_expire=0.000000 has v0 lc 78630
>  -109> 2015-05-18 06:20:08.938432 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f4380 con 0x460a3c0
>  -108> 2015-05-18 06:20:08.938705 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3478599268 0 0) 0x4648900 con 0x460a3c0
>  -107> 2015-05-18 06:20:08.938717 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938718 lease_expire=0.000000 has v0 lc 78630
>  -106> 2015-05-18 06:20:08.938852 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2ac0 con 0x460a3c0
>  -105> 2015-05-18 06:20:08.939109 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466b000 con 0x460a3c0
>  -104> 2015-05-18 06:20:08.939138 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x4648900 con 0x460a3c0
>  -103> 2015-05-18 06:20:08.939153 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8780 con 0x460a3c0
>  -102> 2015-05-18 06:20:08.939168 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466a600 con 0x460a3c0
>  -101> 2015-05-18 06:20:08.939244 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5a00 con 0x460a3c0
>  -100> 2015-05-18 06:20:08.939266 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b81e0 con 0x460a3c0
>   -99> 2015-05-18 06:20:08.939293 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b400 con 0x460a3c0
>   -98> 2015-05-18 06:20:08.939342 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f57c0 con 0x460a3c0
>   -97> 2015-05-18 06:20:08.939354 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a3c0
>   -96> 2015-05-18 06:20:08.949743 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 7 ==== mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 ==== 84+0+0 (110991942 0 0) 0x466b400 con 0x460a3c0
>   -95> 2015-05-18 06:20:08.949801 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1
>   -94> 2015-05-18 06:20:08.949836 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
>   -93> 2015-05-18 06:20:08.949839 7f09817a0700 10 log_client _send_to_monlog to self
>   -92> 2015-05-18 06:20:08.949840 7f09817a0700 10 log_client  log_queue is 1 last_log 20 sent 19 num 1 unsent 1 sending 1
>   -91> 2015-05-18 06:20:08.949843 7f09817a0700 10 log_client  will send 2015-05-18 06:20:08.949838 mon.0 10.8.128.33:6789/0 20 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch
>   -90> 2015-05-18 06:20:08.949851 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5340 con 0x4609600
>   -89> 2015-05-18 06:20:08.949891 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "mon_status", "format": "json"}]=0  v0) v1 -- ?+307 0x4668e00 con 0x460a3c0
>   -88> 2015-05-18 06:20:08.949910 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5340 con 0x4609600
>   -87> 2015-05-18 06:20:08.949919 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.949920 lease_expire=0.000000 has v0 lc 78630
>   -86> 2015-05-18 06:20:08.951317 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 8 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x466a600 con 0x460a3c0
>   -85> 2015-05-18 06:20:08.951357 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
>   -84> 2015-05-18 06:20:08.951380 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>   -83> 2015-05-18 06:20:08.951382 7f09817a0700 10 log_client _send_to_monlog to self
>   -82> 2015-05-18 06:20:08.951384 7f09817a0700 10 log_client  log_queue is 2 last_log 21 sent 20 num 2 unsent 1 sending 1
>   -81> 2015-05-18 06:20:08.951386 7f09817a0700 10 log_client  will send 2015-05-18 06:20:08.951382 mon.0 10.8.128.33:6789/0 21 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>   -80> 2015-05-18 06:20:08.951394 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5580 con 0x4609600
>   -79> 2015-05-18 06:20:08.951519 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0  v0) v1 -- ?+1108 0x466b400 con 0x460a3c0
>   -78> 2015-05-18 06:20:08.951538 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5580 con 0x4609600
>   -77> 2015-05-18 06:20:08.951548 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.951548 lease_expire=0.000000 has v0 lc 78630
>   -76> 2015-05-18 06:20:08.952793 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 9 ==== mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 ==== 94+0+0 (1843246703 0 0) 0x466b000 con 0x460a3c0
>   -75> 2015-05-18 06:20:08.952829 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1
>   -74> 2015-05-18 06:20:08.952852 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
>   -73> 2015-05-18 06:20:08.952855 7f09817a0700 10 log_client _send_to_monlog to self
>   -72> 2015-05-18 06:20:08.952856 7f09817a0700 10 log_client  log_queue is 3 last_log 22 sent 21 num 3 unsent 1 sending 1
>   -71> 2015-05-18 06:20:08.952858 7f09817a0700 10 log_client  will send 2015-05-18 06:20:08.952854 mon.0 10.8.128.33:6789/0 22 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch
>   -70> 2015-05-18 06:20:08.952864 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600
>   -69> 2015-05-18 06:20:08.952941 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "health", "detail": "", "format": "json"}]=0  v0) v1 -- ?+401 0x466a600 con 0x460a3c0
>   -68> 2015-05-18 06:20:08.952960 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600
>   -67> 2015-05-18 06:20:08.952970 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.952970 lease_expire=0.000000 has v0 lc 78630
>   -66> 2015-05-18 06:20:08.954184 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 10 ==== mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 ==== 112+0+0 (3334637970 0 0) 0x4669800 con 0x460a3c0
>   -65> 2015-05-18 06:20:08.954227 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1
>   -64> 2015-05-18 06:20:08.954251 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
>   -63> 2015-05-18 06:20:08.954255 7f09817a0700 10 log_client _send_to_monlog to self
>   -62> 2015-05-18 06:20:08.954256 7f09817a0700 10 log_client  log_queue is 4 last_log 23 sent 22 num 4 unsent 1 sending 1
>   -61> 2015-05-18 06:20:08.954258 7f09817a0700 10 log_client  will send 2015-05-18 06:20:08.954254 mon.0 10.8.128.33:6789/0 23 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch
>   -60> 2015-05-18 06:20:08.954266 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f60c0 con 0x4609600
>   -59> 2015-05-18 06:20:08.954285 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954286 lease_expire=0.000000 has v0 lc 78630
>   -58> 2015-05-18 06:20:08.954605 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]=0 dumped pgs_brief in format json v568) v1 -- ?+6616 0x466b000 con 0x460a3c0
>   -57> 2015-05-18 06:20:08.954626 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f60c0 con 0x4609600
>   -56> 2015-05-18 06:20:08.954636 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954636 lease_expire=0.000000 has v0 lc 78630
>   -55> 2015-05-18 06:20:08.957218 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes
>   -54> 2015-05-18 06:20:08.965521 7f0984a8b700  1 do_command 'config show' 'format:json 
>   -53> 2015-05-18 06:20:08.970043 7f0984a8b700  1 do_command 'config show' 'format:json result is 26602 bytes
>   -52> 2015-05-18 06:20:08.970050 7f0984a8b700  5 asok(0x4650000) AdminSocket: request 'config show' '' to 0x45c0050 returned 26602 bytes
>   -51> 2015-05-18 06:20:09.029915 7f097f69b700  2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).reader couldn't read tag, (0) Success
>   -50> 2015-05-18 06:20:09.029958 7f097f69b700  2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).fault (0) Success
>   -49> 2015-05-18 06:20:09.134600 7f097f69b700  1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460aaa0).accept sd=21 10.8.128.33:60119/0
>   -48> 2015-05-18 06:20:09.134790 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f5a00 con 0x460aaa0
>   -47> 2015-05-18 06:20:09.134816 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.134817 lease_expire=0.000000 has v0 lc 78631
>   -46> 2015-05-18 06:20:09.134841 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0
>   -45> 2015-05-18 06:20:09.134870 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0
>   -44> 2015-05-18 06:20:09.135204 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2513201549 0 0) 0x47f7980 con 0x460aaa0
>   -43> 2015-05-18 06:20:09.135222 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135223 lease_expire=0.000000 has v0 lc 78631
>   -42> 2015-05-18 06:20:09.135370 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f5a00 con 0x460aaa0
>   -41> 2015-05-18 06:20:09.135590 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (692588905 0 0) 0x47f7740 con 0x460aaa0
>   -40> 2015-05-18 06:20:09.135605 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135606 lease_expire=0.000000 has v0 lc 78631
>   -39> 2015-05-18 06:20:09.135738 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0
>   -38> 2015-05-18 06:20:09.135936 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466a600 con 0x460aaa0
>   -37> 2015-05-18 06:20:09.135967 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0
>   -36> 2015-05-18 06:20:09.135984 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b85a0 con 0x460aaa0
>   -35> 2015-05-18 06:20:09.136000 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x4668e00 con 0x460aaa0
>   -34> 2015-05-18 06:20:09.136101 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2ac0 con 0x460aaa0
>   -33> 2015-05-18 06:20:09.136120 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b90e0 con 0x460aaa0
>   -32> 2015-05-18 06:20:09.136139 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b000 con 0x460aaa0
>   -31> 2015-05-18 06:20:09.136183 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5e80 con 0x460aaa0
>   -30> 2015-05-18 06:20:09.136195 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b92c0 con 0x460aaa0
>   -29> 2015-05-18 06:20:09.143538 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 7 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x4669800 con 0x460aaa0
>   -28> 2015-05-18 06:20:09.143603 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1
>   -27> 2015-05-18 06:20:09.143635 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>   -26> 2015-05-18 06:20:09.143638 7f09817a0700 10 log_client _send_to_monlog to self
>   -25> 2015-05-18 06:20:09.143639 7f09817a0700 10 log_client  log_queue is 5 last_log 24 sent 23 num 5 unsent 1 sending 1
>   -24> 2015-05-18 06:20:09.143642 7f09817a0700 10 log_client  will send 2015-05-18 06:20:09.143637 mon.0 10.8.128.33:6789/0 24 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch
>   -23> 2015-05-18 06:20:09.143652 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600
>   -22> 2015-05-18 06:20:09.143795 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0  v0) v1 -- ?+1108 0x466b000 con 0x460aaa0
>   -21> 2015-05-18 06:20:09.143817 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600
>   -20> 2015-05-18 06:20:09.143827 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.143828 lease_expire=0.000000 has v0 lc 78631
>   -19> 2015-05-18 06:20:09.145454 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 8 ==== mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 ==== 82+0+0 (3917390200 0 0) 0x466a000 con 0x460aaa0
>   -18> 2015-05-18 06:20:09.145494 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1
>   -17> 2015-05-18 06:20:09.145520 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
>   -16> 2015-05-18 06:20:09.145522 7f09817a0700 10 log_client _send_to_monlog to self
>   -15> 2015-05-18 06:20:09.145523 7f09817a0700 10 log_client  log_queue is 6 last_log 25 sent 24 num 6 unsent 1 sending 1
>   -14> 2015-05-18 06:20:09.145527 7f09817a0700 10 log_client  will send 2015-05-18 06:20:09.145522 mon.0 10.8.128.33:6789/0 25 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch
>   -13> 2015-05-18 06:20:09.145535 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f6540 con 0x4609600
>   -12> 2015-05-18 06:20:09.145546 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145547 lease_expire=0.000000 has v0 lc 78631
>   -11> 2015-05-18 06:20:09.145779 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0  v85) v1 -- ?+3436 0x4669800 con 0x460aaa0
>   -10> 2015-05-18 06:20:09.145805 7f09817a0700  1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f6540 con 0x4609600
>    -9> 2015-05-18 06:20:09.145814 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145815 lease_expire=0.000000 has v0 lc 78631
>    -8> 2015-05-18 06:20:09.147249 7f09817a0700  1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 9 ==== mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 ==== 95+0+0 (942654298 0 0) 0x466b800 con 0x460aaa0
>    -7> 2015-05-18 06:20:09.147309 7f09817a0700  0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1
>    -6> 2015-05-18 06:20:09.147337 7f09817a0700  0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
>    -5> 2015-05-18 06:20:09.147343 7f09817a0700 10 log_client _send_to_monlog to self
>    -4> 2015-05-18 06:20:09.147345 7f09817a0700 10 log_client  log_queue is 7 last_log 26 sent 25 num 7 unsent 1 sending 1
>    -3> 2015-05-18 06:20:09.147347 7f09817a0700 10 log_client  will send 2015-05-18 06:20:09.147343 mon.0 10.8.128.33:6789/0 26 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch
>    -2> 2015-05-18 06:20:09.147355 7f09817a0700  1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600
>    -1> 2015-05-18 06:20:09.147365 7f09817a0700  5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.147366 lease_expire=0.000000 has v0 lc 78631
>     0> 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) **
> in thread 7f09817a0700
>
> ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
> 1: /usr/bin/ceph-mon() [0x9017e2]
> 2: (()+0xf130) [0x7f098826e130]
> 3: (gsignal()+0x37) [0x7f0986c885d7]
> 4: (abort()+0x148) [0x7f0986c89cc8]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5]
> 6: (()+0x5e926) [0x7f098758a926]
> 7: (()+0x5e953) [0x7f098758a953]
> 8: (()+0x5eb73) [0x7f098758ab73]
> 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717]
> 10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561]
> 11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918]
> 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9]
> 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9]
> 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5]
> 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b]
> 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3]
> 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c]
> 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89]
> 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936]
> 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403]
> 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a]
> 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d]
> 23: (()+0x7df5) [0x7f0988266df5]
> 24: (clone()+0x6d) [0x7f0986d491ad]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>--- logging levels ---
>   0/ 5 none
>   0/ 1 lockdep
>   0/ 1 context
>   1/ 1 crush
>   1/ 5 mds
>   1/ 5 mds_balancer
>   1/ 5 mds_locker
>   1/ 5 mds_log
>   1/ 5 mds_log_expire
>   1/ 5 mds_migrator
>   0/ 1 buffer
>   0/ 1 timer
>   0/ 1 filer
>   0/ 1 striper
>   0/ 1 objecter
>   0/ 5 rados
>   0/ 5 rbd
>   0/ 5 rbd_replay
>   0/ 5 journaler
>   0/ 5 objectcacher
>   0/ 5 client
>   0/ 5 osd
>   0/ 5 optracker
>   0/ 5 objclass
>   1/ 3 filestore
>   1/ 3 keyvaluestore
>   1/ 3 journal
>   0/ 5 ms
>   1/ 5 mon
>   0/10 monc
>   1/ 5 paxos
>   0/ 5 tp
>   1/ 5 auth
>   1/ 5 crypto
>   1/ 1 finisher
>   1/ 5 heartbeatmap
>   1/ 5 perfcounter
>   1/ 5 rgw
>   1/10 civetweb
>   1/ 5 javaclient
>   1/ 5 asok
>   1/ 1 throttle
>   0/ 0 refs
>   1/ 5 xio
>  -2/-2 (syslog threshold)
>  -1/-1 (stderr threshold)
>  max_recent     10000
>  max_new         1000
>  log_file /var/log/ceph/ceph-mon.Mon.log
>--- end dump of recent events ---
>
>===================================
>
>

Actions: View

Attachments on bug 1222509: 1026671 | 1029386 | 1029400 | 1029402