Login
Log in using an SSO provider:
Fedora Account System
Red Hat Associate
Red Hat Customer
Login using a Red Hat Bugzilla account
Forgot Password
Create an Account
Red Hat Bugzilla – Attachment 1026671 Details for
Bug 1222509
1.3.0: mon fails to come up. dies as soon as started.
Home
New
Search
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh92 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
[?]
This site requires JavaScript to be enabled to function correctly, please enable it.
mon log
file_1222509.txt (text/plain), 82.98 KB, created by
Harish NV Rao
on 2015-05-18 11:53:05 UTC
(
hide
)
Description:
mon log
Filename:
MIME Type:
Creator:
Harish NV Rao
Created:
2015-05-18 11:53:05 UTC
Size:
82.98 KB
patch
obsolete
>2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992 >2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 >2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 >2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1 >2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap >2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map >epoch 1 >flags 0 >created 0.000000 >modified 2015-05-06 15:39:42.874456 >tableserver 0 >root 0 >session_timeout 0 >session_autoclose 0 >max_file_size 0 >last_failure 0 >last_failure_osd_epoch 0 >compat compat={},rocompat={},incompat={} >max_mds 0 >in >up {} >failed >stopped >data_pools >metadata_pool 0 >inline_data disabled > >2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires >2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires >2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires >2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires >2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1 >2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1) >2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election >2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0 >2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} >2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail >2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up >2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in >2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch >2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished >2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch >2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished >2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down >2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) >2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 >2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 >2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down >2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) >2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in >2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in >2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail >2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 >2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch >2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 >2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch >2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 >2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch >2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 >2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch >2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 >2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch >2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 >2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch >2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 >2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch >2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) ** > in thread 7f09817a0700 > > ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) > 1: /usr/bin/ceph-mon() [0x9017e2] > 2: (()+0xf130) [0x7f098826e130] > 3: (gsignal()+0x37) [0x7f0986c885d7] > 4: (abort()+0x148) [0x7f0986c89cc8] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5] > 6: (()+0x5e926) [0x7f098758a926] > 7: (()+0x5e953) [0x7f098758a953] > 8: (()+0x5eb73) [0x7f098758ab73] > 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717] > 10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561] > 11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918] > 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9] > 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9] > 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5] > 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b] > 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3] > 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c] > 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89] > 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936] > 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403] > 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a] > 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d] > 23: (()+0x7df5) [0x7f0988266df5] > 24: (clone()+0x6d) [0x7f0986d491ad] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- > -448> 2015-05-18 06:20:01.642177 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_dump hook 0x45c0050 > -447> 2015-05-18 06:20:01.642205 7f09890a87c0 5 asok(0x4650000) register_command 1 hook 0x45c0050 > -446> 2015-05-18 06:20:01.642212 7f09890a87c0 5 asok(0x4650000) register_command perf dump hook 0x45c0050 > -445> 2015-05-18 06:20:01.642219 7f09890a87c0 5 asok(0x4650000) register_command perfcounters_schema hook 0x45c0050 > -444> 2015-05-18 06:20:01.642224 7f09890a87c0 5 asok(0x4650000) register_command 2 hook 0x45c0050 > -443> 2015-05-18 06:20:01.642228 7f09890a87c0 5 asok(0x4650000) register_command perf schema hook 0x45c0050 > -442> 2015-05-18 06:20:01.642233 7f09890a87c0 5 asok(0x4650000) register_command perf reset hook 0x45c0050 > -441> 2015-05-18 06:20:01.642237 7f09890a87c0 5 asok(0x4650000) register_command config show hook 0x45c0050 > -440> 2015-05-18 06:20:01.642242 7f09890a87c0 5 asok(0x4650000) register_command config set hook 0x45c0050 > -439> 2015-05-18 06:20:01.642247 7f09890a87c0 5 asok(0x4650000) register_command config get hook 0x45c0050 > -438> 2015-05-18 06:20:01.642251 7f09890a87c0 5 asok(0x4650000) register_command config diff hook 0x45c0050 > -437> 2015-05-18 06:20:01.642255 7f09890a87c0 5 asok(0x4650000) register_command log flush hook 0x45c0050 > -436> 2015-05-18 06:20:01.642260 7f09890a87c0 5 asok(0x4650000) register_command log dump hook 0x45c0050 > -435> 2015-05-18 06:20:01.642264 7f09890a87c0 5 asok(0x4650000) register_command log reopen hook 0x45c0050 > -434> 2015-05-18 06:20:01.644938 7f09890a87c0 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-mon, pid 3992 > -433> 2015-05-18 06:20:01.667692 7f09890a87c0 5 asok(0x4650000) init /var/run/ceph/ceph-mon.Mon.asok > -432> 2015-05-18 06:20:01.667706 7f09890a87c0 5 asok(0x4650000) bind_and_listen /var/run/ceph/ceph-mon.Mon.asok > -431> 2015-05-18 06:20:01.667808 7f09890a87c0 5 asok(0x4650000) register_command 0 hook 0x45b80b8 > -430> 2015-05-18 06:20:01.667817 7f09890a87c0 5 asok(0x4650000) register_command version hook 0x45b80b8 > -429> 2015-05-18 06:20:01.667823 7f09890a87c0 5 asok(0x4650000) register_command git_version hook 0x45b80b8 > -428> 2015-05-18 06:20:01.667828 7f09890a87c0 5 asok(0x4650000) register_command help hook 0x45c00b0 > -427> 2015-05-18 06:20:01.667833 7f09890a87c0 5 asok(0x4650000) register_command get_command_descriptions hook 0x45c0150 > -426> 2015-05-18 06:20:01.667864 7f0984a8b700 5 asok(0x4650000) entry start > -425> 2015-05-18 06:20:01.742013 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 1496 bytes > -424> 2015-05-18 06:20:01.759779 7f09890a87c0 0 starting mon.Mon rank 0 at 10.8.128.33:6789/0 mon_data /var/lib/ceph/mon/ceph-Mon fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 > -423> 2015-05-18 06:20:01.759829 7f09890a87c0 1 -- 10.8.128.33:6789/0 learned my addr 10.8.128.33:6789/0 > -422> 2015-05-18 06:20:01.759835 7f09890a87c0 1 accepter.accepter.bind my_inst.addr is 10.8.128.33:6789/0 need_addr=0 > -421> 2015-05-18 06:20:01.759884 7f09890a87c0 5 adding auth protocol: cephx > -420> 2015-05-18 06:20:01.759890 7f09890a87c0 5 adding auth protocol: cephx > -419> 2015-05-18 06:20:01.759909 7f09890a87c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info) > -418> 2015-05-18 06:20:01.759913 7f09890a87c0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info) > -417> 2015-05-18 06:20:01.759988 7f09890a87c0 1 mon.Mon@-1(probing) e1 preinit fsid 283bbd8b-9205-4cfc-83fa-80546ba2fe36 > -416> 2015-05-18 06:20:01.760303 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(pgmap 1..567) refresh upgraded, format 0 -> 1 > -415> 2015-05-18 06:20:01.760314 7f09890a87c0 1 mon.Mon@-1(probing).pg v0 on_upgrade discarding in-core PGMap > -414> 2015-05-18 06:20:01.760919 7f09890a87c0 4 mon.Mon@-1(probing).mds e1 new map > -413> 2015-05-18 06:20:01.760927 7f09890a87c0 0 mon.Mon@-1(probing).mds e1 print_map >epoch 1 >flags 0 >created 0.000000 >modified 2015-05-06 15:39:42.874456 >tableserver 0 >root 0 >session_timeout 0 >session_autoclose 0 >max_file_size 0 >last_failure 0 >last_failure_osd_epoch 0 >compat compat={},rocompat={},incompat={} >max_mds 0 >in >up {} >failed >stopped >data_pools >metadata_pool 0 >inline_data disabled > > -412> 2015-05-18 06:20:01.761127 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires > -411> 2015-05-18 06:20:01.761135 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires > -410> 2015-05-18 06:20:01.761138 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires > -409> 2015-05-18 06:20:01.761141 7f09890a87c0 0 mon.Mon@-1(probing).osd e84 crush map has features 1073741824, adjusting msgr requires > -408> 2015-05-18 06:20:01.761445 7f09890a87c0 1 mon.Mon@-1(probing).paxosservice(auth 251..328) refresh upgraded, format 0 -> 1 > -407> 2015-05-18 06:20:01.761792 7f09890a87c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-Mon/keyring > -406> 2015-05-18 06:20:01.761799 7f09890a87c0 5 asok(0x4650000) register_command mon_status hook 0x45c0330 > -405> 2015-05-18 06:20:01.761804 7f09890a87c0 5 asok(0x4650000) register_command quorum_status hook 0x45c0330 > -404> 2015-05-18 06:20:01.761808 7f09890a87c0 5 asok(0x4650000) register_command sync_force hook 0x45c0330 > -403> 2015-05-18 06:20:01.761811 7f09890a87c0 5 asok(0x4650000) register_command add_bootstrap_peer_hint hook 0x45c0330 > -402> 2015-05-18 06:20:01.761814 7f09890a87c0 5 asok(0x4650000) register_command quorum enter hook 0x45c0330 > -401> 2015-05-18 06:20:01.761817 7f09890a87c0 5 asok(0x4650000) register_command quorum exit hook 0x45c0330 > -400> 2015-05-18 06:20:01.761825 7f09890a87c0 1 -- 10.8.128.33:6789/0 messenger.start > -399> 2015-05-18 06:20:01.761854 7f09890a87c0 2 mon.Mon@-1(probing) e1 init > -398> 2015-05-18 06:20:01.761912 7f09890a87c0 1 accepter.accepter.start > -397> 2015-05-18 06:20:01.761932 7f09890a87c0 0 mon.Mon@-1(probing) e1 my rank is now 0 (was -1) > -396> 2015-05-18 06:20:01.761939 7f09890a87c0 1 -- 10.8.128.33:6789/0 mark_down_all > -395> 2015-05-18 06:20:01.761953 7f09890a87c0 1 mon.Mon@0(probing) e1 win_standalone_election > -394> 2015-05-18 06:20:01.801222 7f09890a87c0 0 log_channel(cluster) log [INF] : mon.Mon@0 won leader election with quorum 0 > -393> 2015-05-18 06:20:01.801230 7f09890a87c0 10 log_client _send_to_monlog to self > -392> 2015-05-18 06:20:01.801231 7f09890a87c0 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 > -391> 2015-05-18 06:20:01.801235 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0 > -390> 2015-05-18 06:20:01.801247 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 > -389> 2015-05-18 06:20:01.801309 7f09890a87c0 0 log_channel(cluster) log [INF] : monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} > -388> 2015-05-18 06:20:01.801316 7f09890a87c0 10 log_client _send_to_monlog to self > -387> 2015-05-18 06:20:01.801317 7f09890a87c0 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1 > -386> 2015-05-18 06:20:01.801298 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 > -385> 2015-05-18 06:20:01.801320 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} > -384> 2015-05-18 06:20:01.801327 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600 > -383> 2015-05-18 06:20:01.801372 7f09890a87c0 0 log_channel(cluster) log [INF] : pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -382> 2015-05-18 06:20:01.801376 7f09890a87c0 10 log_client _send_to_monlog to self > -381> 2015-05-18 06:20:01.801377 7f09890a87c0 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1 > -380> 2015-05-18 06:20:01.801380 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -379> 2015-05-18 06:20:01.801387 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2880 con 0x4609600 > -378> 2015-05-18 06:20:01.801422 7f09890a87c0 0 log_channel(cluster) log [INF] : mdsmap e1: 0/0/0 up > -377> 2015-05-18 06:20:01.801426 7f09890a87c0 10 log_client _send_to_monlog to self > -376> 2015-05-18 06:20:01.801427 7f09890a87c0 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1 > -375> 2015-05-18 06:20:01.801428 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up > -374> 2015-05-18 06:20:01.801435 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2640 con 0x4609600 > -373> 2015-05-18 06:20:01.801492 7f09890a87c0 0 log_channel(cluster) log [INF] : osdmap e84: 4 osds: 4 up, 4 in > -372> 2015-05-18 06:20:01.801496 7f09890a87c0 10 log_client _send_to_monlog to self > -371> 2015-05-18 06:20:01.801498 7f09890a87c0 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1 > -370> 2015-05-18 06:20:01.801499 7f09890a87c0 10 log_client will send 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in > -369> 2015-05-18 06:20:01.801506 7f09890a87c0 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 > -368> 2015-05-18 06:20:01.801773 7f09817a0700 5 mon.Mon@0(leader) e1 setting monitor caps on this connection > -367> 2015-05-18 06:20:01.801790 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801791 lease_expire=0.000000 has v0 lc 78623 > -366> 2015-05-18 06:20:01.801820 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600 > -365> 2015-05-18 06:20:01.801832 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801833 lease_expire=0.000000 has v0 lc 78623 > -364> 2015-05-18 06:20:01.801846 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2880 con 0x4609600 > -363> 2015-05-18 06:20:01.801853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801854 lease_expire=0.000000 has v0 lc 78623 > -362> 2015-05-18 06:20:01.801866 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2640 con 0x4609600 > -361> 2015-05-18 06:20:01.801874 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801874 lease_expire=0.000000 has v0 lc 78623 > -360> 2015-05-18 06:20:01.801886 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 > -359> 2015-05-18 06:20:01.801894 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) is_readable = 1 - now=2015-05-18 06:20:01.801894 lease_expire=0.000000 has v0 lc 78623 > -358> 2015-05-18 06:20:01.851959 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78623) queue_pending_finisher 0x45c04d0 > -357> 2015-05-18 06:20:01.918852 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 1) v1 -- ?+0 0x47b81e0 con 0x4609600 > -356> 2015-05-18 06:20:01.918871 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 2) v1 -- ?+0 0x47b85a0 con 0x4609600 > -355> 2015-05-18 06:20:01.918879 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 3) v1 -- ?+0 0x47b83c0 con 0x4609600 > -354> 2015-05-18 06:20:01.918886 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 4) v1 -- ?+0 0x47b8b40 con 0x4609600 > -353> 2015-05-18 06:20:01.918894 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 5) v1 -- ?+0 0x47b8960 con 0x4609600 > -352> 2015-05-18 06:20:01.918930 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 > -351> 2015-05-18 06:20:01.918953 7f09817a0700 10 log_client handle_log_ack log(last 1) v1 > -350> 2015-05-18 06:20:01.918956 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801229 mon.0 10.8.128.33:6789/0 1 : cluster [INF] mon.Mon@0 won leader election with quorum 0 > -349> 2015-05-18 06:20:01.918977 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600 > -348> 2015-05-18 06:20:01.918986 7f09817a0700 10 log_client handle_log_ack log(last 2) v1 > -347> 2015-05-18 06:20:01.918989 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801314 mon.0 10.8.128.33:6789/0 2 : cluster [INF] monmap e1: 1 mons at {Mon=10.8.128.33:6789/0} > -346> 2015-05-18 06:20:01.918997 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 > -345> 2015-05-18 06:20:01.919005 7f09817a0700 10 log_client handle_log_ack log(last 3) v1 > -344> 2015-05-18 06:20:01.919007 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801376 mon.0 10.8.128.33:6789/0 3 : cluster [INF] pgmap v567: 64 pgs: 17 stale+active+clean, 47 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -343> 2015-05-18 06:20:01.919015 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 > -342> 2015-05-18 06:20:01.919022 7f09817a0700 10 log_client handle_log_ack log(last 4) v1 > -341> 2015-05-18 06:20:01.919025 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801425 mon.0 10.8.128.33:6789/0 4 : cluster [INF] mdsmap e1: 0/0/0 up > -340> 2015-05-18 06:20:01.919034 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 > -339> 2015-05-18 06:20:01.919041 7f09817a0700 10 log_client handle_log_ack log(last 5) v1 > -338> 2015-05-18 06:20:01.919043 7f09817a0700 10 log_client logged 2015-05-18 06:20:01.801496 mon.0 10.8.128.33:6789/0 5 : cluster [INF] osdmap e84: 4 osds: 4 up, 4 in > -337> 2015-05-18 06:20:02.473956 7f0989044700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48d0000 sd=20 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a260).accept sd=20 10.8.128.86:48717/0 > -336> 2015-05-18 06:20:02.474299 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 1 ==== auth(proto 0 26 bytes epoch 1) v1 ==== 56+0+0 (1708591557 0 0) 0x47f3a80 con 0x460a260 > -335> 2015-05-18 06:20:02.474322 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474323 lease_expire=0.000000 has v0 lc 78624 > -334> 2015-05-18 06:20:02.474362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2f40 con 0x460a260 > -333> 2015-05-18 06:20:02.474629 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1575692119 0 0) 0x47f4140 con 0x460a260 > -332> 2015-05-18 06:20:02.474645 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.474646 lease_expire=0.000000 has v0 lc 78624 > -331> 2015-05-18 06:20:02.474885 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f3a80 con 0x460a260 > -330> 2015-05-18 06:20:02.475200 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 3 ==== mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 ==== 50+0+0 (3889722309 0 0) 0x466a600 con 0x460a260 > -329> 2015-05-18 06:20:02.475267 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a260 > -328> 2015-05-18 06:20:02.475300 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 4 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 765sec e84 v84) v3 ==== 188+0+0 (2601468610 0 0) 0x46798c0 con 0x460a260 > -327> 2015-05-18 06:20:02.475316 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475317 lease_expire=0.000000 has v0 lc 78624 > -326> 2015-05-18 06:20:02.475336 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -325> 2015-05-18 06:20:02.475350 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -324> 2015-05-18 06:20:02.475353 7f09817a0700 10 log_client _send_to_monlog to self > -323> 2015-05-18 06:20:02.475354 7f09817a0700 10 log_client log_queue is 1 last_log 6 sent 5 num 1 unsent 1 sending 1 > -322> 2015-05-18 06:20:02.475356 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -321> 2015-05-18 06:20:02.475362 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600 > -320> 2015-05-18 06:20:02.475385 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 5 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x4679e40 con 0x460a260 > -319> 2015-05-18 06:20:02.475396 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475397 lease_expire=0.000000 has v0 lc 78624 > -318> 2015-05-18 06:20:02.475405 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -317> 2015-05-18 06:20:02.475418 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -316> 2015-05-18 06:20:02.475420 7f09817a0700 10 log_client _send_to_monlog to self > -315> 2015-05-18 06:20:02.475421 7f09817a0700 10 log_client log_queue is 2 last_log 7 sent 6 num 2 unsent 1 sending 1 > -314> 2015-05-18 06:20:02.475423 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -313> 2015-05-18 06:20:02.475428 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2ac0 con 0x4609600 > -312> 2015-05-18 06:20:02.475444 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f3cc0 con 0x4609600 > -311> 2015-05-18 06:20:02.475452 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475453 lease_expire=0.000000 has v0 lc 78624 > -310> 2015-05-18 06:20:02.475470 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2ac0 con 0x4609600 > -309> 2015-05-18 06:20:02.475479 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.475480 lease_expire=0.000000 has v0 lc 78624 > -308> 2015-05-18 06:20:02.475496 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 6 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=84}) v2 ==== 69+0+0 (2997052209 0 0) 0x466b000 con 0x460a260 > -307> 2015-05-18 06:20:02.475517 7f09817a0700 5 mon.Mon@0(leader).osd e84 send_incremental [84..84] to osd.1 10.8.128.86:6800/19549 > -306> 2015-05-18 06:20:02.475587 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..84 src has 1..84) v3 -- ?+0 0x47f4140 con 0x460a260 > -305> 2015-05-18 06:20:02.475605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b83c0 con 0x460a260 > -304> 2015-05-18 06:20:02.815632 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes > -303> 2015-05-18 06:20:02.823300 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch > -302> 2015-05-18 06:20:02.823308 7f0984a8b700 10 log_client _send_to_monlog to self > -301> 2015-05-18 06:20:02.823309 7f0984a8b700 10 log_client log_queue is 3 last_log 8 sent 7 num 3 unsent 1 sending 1 > -300> 2015-05-18 06:20:02.823312 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch > -299> 2015-05-18 06:20:02.823333 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648d80 con 0x4609600 > -298> 2015-05-18 06:20:02.823374 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished > -297> 2015-05-18 06:20:02.823377 7f0984a8b700 10 log_client _send_to_monlog to self > -296> 2015-05-18 06:20:02.823378 7f0984a8b700 10 log_client log_queue is 4 last_log 9 sent 8 num 4 unsent 1 sending 1 > -295> 2015-05-18 06:20:02.823380 7f0984a8b700 10 log_client will send 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished > -294> 2015-05-18 06:20:02.823386 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600 > -293> 2015-05-18 06:20:02.823393 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648d80 con 0x4609600 > -292> 2015-05-18 06:20:02.823407 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 520 bytes > -291> 2015-05-18 06:20:02.823421 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823422 lease_expire=0.000000 has v0 lc 78624 > -290> 2015-05-18 06:20:02.823450 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600 > -289> 2015-05-18 06:20:02.823467 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.823468 lease_expire=0.000000 has v0 lc 78624 > -288> 2015-05-18 06:20:02.832644 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 7 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 766sec e84 v84) v3 ==== 188+0+0 (4180528123 0 0) 0x4679b80 con 0x460a260 > -287> 2015-05-18 06:20:02.832671 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832672 lease_expire=0.000000 has v0 lc 78624 > -286> 2015-05-18 06:20:02.832689 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -285> 2015-05-18 06:20:02.832704 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -284> 2015-05-18 06:20:02.832707 7f09817a0700 10 log_client _send_to_monlog to self > -283> 2015-05-18 06:20:02.832708 7f09817a0700 10 log_client log_queue is 5 last_log 10 sent 9 num 5 unsent 1 sending 1 > -282> 2015-05-18 06:20:02.832710 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -281> 2015-05-18 06:20:02.832717 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 > -280> 2015-05-18 06:20:02.832739 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 8 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 723sec e84 v84) v3 ==== 188+0+0 (967876087 0 0) 0x467a680 con 0x460a260 > -279> 2015-05-18 06:20:02.832751 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832751 lease_expire=0.000000 has v0 lc 78624 > -278> 2015-05-18 06:20:02.832760 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -277> 2015-05-18 06:20:02.832773 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -276> 2015-05-18 06:20:02.832775 7f09817a0700 10 log_client _send_to_monlog to self > -275> 2015-05-18 06:20:02.832777 7f09817a0700 10 log_client log_queue is 6 last_log 11 sent 10 num 6 unsent 1 sending 1 > -274> 2015-05-18 06:20:02.832779 7f09817a0700 10 log_client will send 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -273> 2015-05-18 06:20:02.832785 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600 > -272> 2015-05-18 06:20:02.832807 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 > -271> 2015-05-18 06:20:02.832817 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832817 lease_expire=0.000000 has v0 lc 78624 > -270> 2015-05-18 06:20:02.832833 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600 > -269> 2015-05-18 06:20:02.832842 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) is_readable = 1 - now=2015-05-18 06:20:02.832844 lease_expire=0.000000 has v0 lc 78624 > -268> 2015-05-18 06:20:02.918393 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78624) queue_pending_finisher 0x45c04c0 > -267> 2015-05-18 06:20:02.977214 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 6) v1 -- ?+0 0x47b8780 con 0x4609600 > -266> 2015-05-18 06:20:02.977246 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 7) v1 -- ?+0 0x47b81e0 con 0x4609600 > -265> 2015-05-18 06:20:02.977254 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 8) v1 -- ?+0 0x47b8b40 con 0x4609600 > -264> 2015-05-18 06:20:02.977259 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 9) v1 -- ?+0 0x47b83c0 con 0x4609600 > -263> 2015-05-18 06:20:02.977267 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 10) v1 -- ?+0 0x47b8960 con 0x4609600 > -262> 2015-05-18 06:20:02.977291 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 11) v1 -- ?+0 0x47b92c0 con 0x4609600 > -261> 2015-05-18 06:20:02.977297 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 6) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600 > -260> 2015-05-18 06:20:02.977319 7f09817a0700 10 log_client handle_log_ack log(last 6) v1 > -259> 2015-05-18 06:20:02.977322 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475352 mon.0 10.8.128.33:6789/0 6 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -258> 2015-05-18 06:20:02.977334 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 > -257> 2015-05-18 06:20:02.977341 7f09817a0700 10 log_client handle_log_ack log(last 7) v1 > -256> 2015-05-18 06:20:02.977343 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.475419 mon.0 10.8.128.33:6789/0 7 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -255> 2015-05-18 06:20:02.977352 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 8) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 > -254> 2015-05-18 06:20:02.977375 7f09817a0700 10 log_client handle_log_ack log(last 8) v1 > -253> 2015-05-18 06:20:02.977379 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823306 mon.0 10.8.128.33:6789/0 8 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch > -252> 2015-05-18 06:20:02.977388 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 9) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 > -251> 2015-05-18 06:20:02.977395 7f09817a0700 10 log_client handle_log_ack log(last 9) v1 > -250> 2015-05-18 06:20:02.977398 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.823376 mon.0 10.8.128.33:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished > -249> 2015-05-18 06:20:02.977405 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 10) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 > -248> 2015-05-18 06:20:02.977412 7f09817a0700 10 log_client handle_log_ack log(last 10) v1 > -247> 2015-05-18 06:20:02.977414 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832706 mon.0 10.8.128.33:6789/0 10 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -246> 2015-05-18 06:20:02.977422 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 11) v1 ==== 0+0+0 (0 0 0) 0x47b92c0 con 0x4609600 > -245> 2015-05-18 06:20:02.977430 7f09817a0700 10 log_client handle_log_ack log(last 11) v1 > -244> 2015-05-18 06:20:02.977431 7f09817a0700 10 log_client logged 2015-05-18 06:20:02.832774 mon.0 10.8.128.33:6789/0 11 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -243> 2015-05-18 06:20:06.761957 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761960 lease_expire=0.000000 has v0 lc 78625 > -242> 2015-05-18 06:20:06.761973 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761974 lease_expire=0.000000 has v0 lc 78625 > -241> 2015-05-18 06:20:06.761993 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) is_readable = 1 - now=2015-05-18 06:20:06.761993 lease_expire=0.000000 has v0 lc 78625 > -240> 2015-05-18 06:20:06.762078 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78625) queue_pending_finisher 0x45c04f0 > -239> 2015-05-18 06:20:07.793435 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes > -238> 2015-05-18 06:20:07.802963 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch > -237> 2015-05-18 06:20:07.802972 7f0984a8b700 10 log_client _send_to_monlog to self > -236> 2015-05-18 06:20:07.802973 7f0984a8b700 10 log_client log_queue is 1 last_log 12 sent 11 num 1 unsent 1 sending 1 > -235> 2015-05-18 06:20:07.802977 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch > -234> 2015-05-18 06:20:07.802995 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x4648900 con 0x4609600 > -233> 2015-05-18 06:20:07.803031 7f0984a8b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished > -232> 2015-05-18 06:20:07.803033 7f0984a8b700 10 log_client _send_to_monlog to self > -231> 2015-05-18 06:20:07.803034 7f0984a8b700 10 log_client log_queue is 2 last_log 13 sent 12 num 2 unsent 1 sending 1 > -230> 2015-05-18 06:20:07.803037 7f0984a8b700 10 log_client will send 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished > -229> 2015-05-18 06:20:07.803043 7f0984a8b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 > -228> 2015-05-18 06:20:07.803059 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'mon_status' '' to 0x45c0330 returned 307 bytes > -227> 2015-05-18 06:20:07.803053 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4648900 con 0x4609600 > -226> 2015-05-18 06:20:07.803083 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803084 lease_expire=0.000000 has v0 lc 78626 > -225> 2015-05-18 06:20:07.803114 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 > -224> 2015-05-18 06:20:07.803123 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.803125 lease_expire=0.000000 has v0 lc 78626 > -223> 2015-05-18 06:20:07.803590 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes > -222> 2015-05-18 06:20:07.810261 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78626) queue_pending_finisher 0x45c0320 > -221> 2015-05-18 06:20:07.812704 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'version' '' to 0x45b80b8 returned 20 bytes > -220> 2015-05-18 06:20:07.833354 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 9 ==== osd_failure(failed osd.0 10.8.128.40:6800/25974 for 771sec e84 v84) v3 ==== 188+0+0 (3406336582 0 0) 0x467a3c0 con 0x460a260 > -219> 2015-05-18 06:20:07.835570 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835574 lease_expire=0.000000 has v0 lc 78626 > -218> 2015-05-18 06:20:07.835601 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.0 10.8.128.40:6800/25974 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -217> 2015-05-18 06:20:07.835627 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -216> 2015-05-18 06:20:07.835632 7f09817a0700 10 log_client _send_to_monlog to self > -215> 2015-05-18 06:20:07.835633 7f09817a0700 10 log_client log_queue is 3 last_log 14 sent 13 num 3 unsent 1 sending 1 > -214> 2015-05-18 06:20:07.835639 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -213> 2015-05-18 06:20:07.835652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4ec0 con 0x4609600 > -212> 2015-05-18 06:20:07.835680 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.0 down > -211> 2015-05-18 06:20:07.835700 7f09817a0700 0 log_channel(cluster) log [INF] : osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) > -210> 2015-05-18 06:20:07.835703 7f09817a0700 10 log_client _send_to_monlog to self > -209> 2015-05-18 06:20:07.835704 7f09817a0700 10 log_client log_queue is 4 last_log 15 sent 14 num 4 unsent 1 sending 1 > -208> 2015-05-18 06:20:07.835707 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) > -207> 2015-05-18 06:20:07.835722 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4c80 con 0x4609600 > -206> 2015-05-18 06:20:07.835744 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4ec0 con 0x4609600 > -205> 2015-05-18 06:20:07.835759 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835761 lease_expire=0.000000 has v0 lc 78626 > -204> 2015-05-18 06:20:07.835780 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4c80 con 0x4609600 > -203> 2015-05-18 06:20:07.835809 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835810 lease_expire=0.000000 has v0 lc 78626 > -202> 2015-05-18 06:20:07.835826 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 10 ==== osd_failure(failed osd.3 10.8.128.86:6804/19849 for 728sec e84 v84) v3 ==== 188+0+0 (2916513576 0 0) 0x467a100 con 0x460a260 > -201> 2015-05-18 06:20:07.835853 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835854 lease_expire=0.000000 has v0 lc 78626 > -200> 2015-05-18 06:20:07.835868 7f09817a0700 1 mon.Mon@0(leader).osd e84 prepare_failure osd.3 10.8.128.86:6804/19849 from osd.1 10.8.128.86:6800/19549 is reporting failure:1 > -199> 2015-05-18 06:20:07.835888 7f09817a0700 0 log_channel(cluster) log [DBG] : osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -198> 2015-05-18 06:20:07.835892 7f09817a0700 10 log_client _send_to_monlog to self > -197> 2015-05-18 06:20:07.835893 7f09817a0700 10 log_client log_queue is 5 last_log 16 sent 15 num 5 unsent 1 sending 1 > -196> 2015-05-18 06:20:07.835897 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -195> 2015-05-18 06:20:07.835907 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4a40 con 0x4609600 > -194> 2015-05-18 06:20:07.835925 7f09817a0700 1 mon.Mon@0(leader).osd e84 we have enough reports/reporters to mark osd.3 down > -193> 2015-05-18 06:20:07.835939 7f09817a0700 0 log_channel(cluster) log [INF] : osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) > -192> 2015-05-18 06:20:07.835942 7f09817a0700 10 log_client _send_to_monlog to self > -191> 2015-05-18 06:20:07.835943 7f09817a0700 10 log_client log_queue is 6 last_log 17 sent 16 num 6 unsent 1 sending 1 > -190> 2015-05-18 06:20:07.835946 7f09817a0700 10 log_client will send 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) > -189> 2015-05-18 06:20:07.835956 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5c40 con 0x4609600 > -188> 2015-05-18 06:20:07.835984 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4a40 con 0x4609600 > -187> 2015-05-18 06:20:07.835997 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.835998 lease_expire=0.000000 has v0 lc 78626 > -186> 2015-05-18 06:20:07.836011 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5c40 con 0x4609600 > -185> 2015-05-18 06:20:07.836021 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78626) is_readable = 1 - now=2015-05-18 06:20:07.836022 lease_expire=0.000000 has v0 lc 78626 > -184> 2015-05-18 06:20:07.860966 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 12) v1 -- ?+0 0x47b90e0 con 0x4609600 > -183> 2015-05-18 06:20:07.860985 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 13) v1 -- ?+0 0x47b8f00 con 0x4609600 > -182> 2015-05-18 06:20:07.860993 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.860994 lease_expire=0.000000 has v0 lc 78627 > -181> 2015-05-18 06:20:07.861010 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861011 lease_expire=0.000000 has v0 lc 78627 > -180> 2015-05-18 06:20:07.861023 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861024 lease_expire=0.000000 has v0 lc 78627 > -179> 2015-05-18 06:20:07.861034 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) is_readable = 1 - now=2015-05-18 06:20:07.861035 lease_expire=0.000000 has v0 lc 78627 > -178> 2015-05-18 06:20:07.861029 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 12) v1 ==== 0+0+0 (0 0 0) 0x47b90e0 con 0x4609600 > -177> 2015-05-18 06:20:07.861052 7f09817a0700 10 log_client handle_log_ack log(last 12) v1 > -176> 2015-05-18 06:20:07.861057 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.802970 mon.0 10.8.128.33:6789/0 12 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch > -175> 2015-05-18 06:20:07.861074 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 13) v1 ==== 0+0+0 (0 0 0) 0x47b8f00 con 0x4609600 > -174> 2015-05-18 06:20:07.861086 7f09817a0700 10 log_client handle_log_ack log(last 13) v1 > -173> 2015-05-18 06:20:07.861090 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.803032 mon.0 10.8.128.33:6789/0 13 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished > -172> 2015-05-18 06:20:07.885804 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.0 DOWN > -171> 2015-05-18 06:20:07.885814 7f0981fa1700 2 mon.Mon@0(leader).osd e84 osd.3 DOWN > -170> 2015-05-18 06:20:07.885893 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78627) queue_pending_finisher 0x45c04d0 > -169> 2015-05-18 06:20:07.937432 7f098370b700 1 mon.Mon@0(leader).osd e85 e85: 4 osds: 2 up, 4 in > -168> 2015-05-18 06:20:07.962214 7f098370b700 5 mon.Mon@0(leader).paxos(paxos refresh c 78062..78628) is_readable = 0 - now=2015-05-18 06:20:07.962218 lease_expire=0.000000 has v0 lc 78628 > -167> 2015-05-18 06:20:07.962309 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(84..85 src has 1..85) v3 -- ?+0 0x47f2400 con 0x460a260 > -166> 2015-05-18 06:20:07.962334 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84 > -165> 2015-05-18 06:20:07.962341 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549 > -164> 2015-05-18 06:20:07.962379 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2f40 con 0x460a260 > -163> 2015-05-18 06:20:07.962394 7f098370b700 5 mon.Mon@0(leader).osd e85 send_latest to osd.1 10.8.128.86:6800/19549 start 84 > -162> 2015-05-18 06:20:07.962399 7f098370b700 5 mon.Mon@0(leader).osd e85 send_incremental [84..85] to osd.1 10.8.128.86:6800/19549 > -161> 2015-05-18 06:20:07.962590 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.962592 lease_expire=0.000000 has v0 lc 78628 > -160> 2015-05-18 06:20:07.962922 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78628) queue_pending_finisher 0x45c04d0 > -159> 2015-05-18 06:20:07.987480 7f098370b700 0 log_channel(cluster) log [INF] : osdmap e85: 4 osds: 2 up, 4 in > -158> 2015-05-18 06:20:07.987489 7f098370b700 10 log_client _send_to_monlog to self > -157> 2015-05-18 06:20:07.987492 7f098370b700 10 log_client log_queue is 5 last_log 18 sent 17 num 5 unsent 1 sending 1 > -156> 2015-05-18 06:20:07.987495 7f098370b700 10 log_client will send 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in > -155> 2015-05-18 06:20:07.987506 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f4800 con 0x4609600 > -154> 2015-05-18 06:20:07.987564 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f4800 con 0x4609600 > -153> 2015-05-18 06:20:07.987582 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78628) is_readable = 1 - now=2015-05-18 06:20:07.987583 lease_expire=0.000000 has v0 lc 78628 > -152> 2015-05-18 06:20:08.005319 7f09817a0700 1 -- 10.8.128.33:6789/0 <== osd.1 10.8.128.86:6800/19549 11 ==== mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=86}) v2 ==== 69+0+0 (1597159760 0 0) 0x466b400 con 0x460a260 > -151> 2015-05-18 06:20:08.005352 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.86:6800/19549 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8f00 con 0x460a260 > -150> 2015-05-18 06:20:08.013617 7f098370b700 0 log_channel(cluster) log [INF] : pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -149> 2015-05-18 06:20:08.013624 7f098370b700 10 log_client _send_to_monlog to self > -148> 2015-05-18 06:20:08.013625 7f098370b700 10 log_client log_queue is 6 last_log 19 sent 18 num 6 unsent 1 sending 1 > -147> 2015-05-18 06:20:08.013627 7f098370b700 10 log_client will send 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -146> 2015-05-18 06:20:08.013637 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5100 con 0x4609600 > -145> 2015-05-18 06:20:08.013689 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5100 con 0x4609600 > -144> 2015-05-18 06:20:08.013702 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.013703 lease_expire=0.000000 has v0 lc 78629 > -143> 2015-05-18 06:20:08.860774 7f0981fa1700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78629) queue_pending_finisher 0x45c0510 > -142> 2015-05-18 06:20:08.909048 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460a3c0).accept sd=21 10.8.128.33:60116/0 > -141> 2015-05-18 06:20:08.909248 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f4140 con 0x460a3c0 > -140> 2015-05-18 06:20:08.909277 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) is_readable = 1 - now=2015-05-18 06:20:08.909278 lease_expire=0.000000 has v0 lc 78629 > -139> 2015-05-18 06:20:08.909363 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78629) queue_pending_finisher 0x45c0500 > -138> 2015-05-18 06:20:08.913369 7f098370b700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.913371 lease_expire=0.000000 has v0 lc 78630 > -137> 2015-05-18 06:20:08.913393 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x47f2ac0 con 0x460a3c0 > -136> 2015-05-18 06:20:08.913421 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4648900 con 0x460a3c0 > -135> 2015-05-18 06:20:08.913486 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 14) v1 -- ?+0 0x47b85a0 con 0x4609600 > -134> 2015-05-18 06:20:08.913503 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 15) v1 -- ?+0 0x47b8960 con 0x4609600 > -133> 2015-05-18 06:20:08.913512 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 16) v1 -- ?+0 0x47b83c0 con 0x4609600 > -132> 2015-05-18 06:20:08.913519 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 17) v1 -- ?+0 0x47b8b40 con 0x4609600 > -131> 2015-05-18 06:20:08.913528 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 18) v1 -- ?+0 0x47b81e0 con 0x4609600 > -130> 2015-05-18 06:20:08.913536 7f098370b700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(last 19) v1 -- ?+0 0x47b8780 con 0x4609600 > -129> 2015-05-18 06:20:08.913547 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 14) v1 ==== 0+0+0 (0 0 0) 0x47b85a0 con 0x4609600 > -128> 2015-05-18 06:20:08.938024 7f09817a0700 10 log_client handle_log_ack log(last 14) v1 > -127> 2015-05-18 06:20:08.938034 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835630 mon.0 10.8.128.33:6789/0 14 : cluster [DBG] osd.0 10.8.128.40:6800/25974 reported failed by osd.1 10.8.128.86:6800/19549 > -126> 2015-05-18 06:20:08.938051 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 15) v1 ==== 0+0+0 (0 0 0) 0x47b8960 con 0x4609600 > -125> 2015-05-18 06:20:08.938060 7f09817a0700 10 log_client handle_log_ack log(last 15) v1 > -124> 2015-05-18 06:20:08.938063 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835702 mon.0 10.8.128.33:6789/0 15 : cluster [INF] osd.0 10.8.128.40:6800/25974 failed (3 reports from 1 peers after 770.360385 >= grace 30.126211) > -123> 2015-05-18 06:20:08.938071 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 16) v1 ==== 0+0+0 (0 0 0) 0x47b83c0 con 0x4609600 > -122> 2015-05-18 06:20:08.938079 7f09817a0700 10 log_client handle_log_ack log(last 16) v1 > -121> 2015-05-18 06:20:08.938083 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835891 mon.0 10.8.128.33:6789/0 16 : cluster [DBG] osd.3 10.8.128.86:6804/19849 reported failed by osd.1 10.8.128.86:6800/19549 > -120> 2015-05-18 06:20:08.938092 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 17) v1 ==== 0+0+0 (0 0 0) 0x47b8b40 con 0x4609600 > -119> 2015-05-18 06:20:08.938099 7f09817a0700 10 log_client handle_log_ack log(last 17) v1 > -118> 2015-05-18 06:20:08.938101 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.835941 mon.0 10.8.128.33:6789/0 17 : cluster [INF] osd.3 10.8.128.86:6804/19849 failed (3 reports from 1 peers after 728.360632 >= grace 307.946015) > -117> 2015-05-18 06:20:08.938109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 18) v1 ==== 0+0+0 (0 0 0) 0x47b81e0 con 0x4609600 > -116> 2015-05-18 06:20:08.938116 7f09817a0700 10 log_client handle_log_ack log(last 18) v1 > -115> 2015-05-18 06:20:08.938119 7f09817a0700 10 log_client logged 2015-05-18 06:20:07.987487 mon.0 10.8.128.33:6789/0 18 : cluster [INF] osdmap e85: 4 osds: 2 up, 4 in > -114> 2015-05-18 06:20:08.938127 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(last 19) v1 ==== 0+0+0 (0 0 0) 0x47b8780 con 0x4609600 > -113> 2015-05-18 06:20:08.938135 7f09817a0700 10 log_client handle_log_ack log(last 19) v1 > -112> 2015-05-18 06:20:08.938137 7f09817a0700 10 log_client logged 2015-05-18 06:20:08.013623 mon.0 10.8.128.33:6789/0 19 : cluster [INF] pgmap v568: 64 pgs: 49 stale+active+clean, 15 active+clean; 0 bytes data, 80141 MB used, 3645 GB / 3724 GB avail > -111> 2015-05-18 06:20:08.938145 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (309592415 0 0) 0x47f2ac0 con 0x460a3c0 > -110> 2015-05-18 06:20:08.938154 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938155 lease_expire=0.000000 has v0 lc 78630 > -109> 2015-05-18 06:20:08.938432 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f4380 con 0x460a3c0 > -108> 2015-05-18 06:20:08.938705 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (3478599268 0 0) 0x4648900 con 0x460a3c0 > -107> 2015-05-18 06:20:08.938717 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.938718 lease_expire=0.000000 has v0 lc 78630 > -106> 2015-05-18 06:20:08.938852 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f2ac0 con 0x460a3c0 > -105> 2015-05-18 06:20:08.939109 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004034 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466b000 con 0x460a3c0 > -104> 2015-05-18 06:20:08.939138 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_map magic: 0 v1 -- ?+0 0x4648900 con 0x460a3c0 > -103> 2015-05-18 06:20:08.939153 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8780 con 0x460a3c0 > -102> 2015-05-18 06:20:08.939168 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466a600 con 0x460a3c0 > -101> 2015-05-18 06:20:08.939244 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5a00 con 0x460a3c0 > -100> 2015-05-18 06:20:08.939266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b81e0 con 0x460a3c0 > -99> 2015-05-18 06:20:08.939293 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b400 con 0x460a3c0 > -98> 2015-05-18 06:20:08.939342 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f57c0 con 0x460a3c0 > -97> 2015-05-18 06:20:08.939354 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b8960 con 0x460a3c0 > -96> 2015-05-18 06:20:08.949743 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 7 ==== mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 ==== 84+0+0 (110991942 0 0) 0x466b400 con 0x460a3c0 > -95> 2015-05-18 06:20:08.949801 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "mon_status", "format": "json"} v 0) v1 > -94> 2015-05-18 06:20:08.949836 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch > -93> 2015-05-18 06:20:08.949839 7f09817a0700 10 log_client _send_to_monlog to self > -92> 2015-05-18 06:20:08.949840 7f09817a0700 10 log_client log_queue is 1 last_log 20 sent 19 num 1 unsent 1 sending 1 > -91> 2015-05-18 06:20:08.949843 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.949838 mon.0 10.8.128.33:6789/0 20 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "mon_status", "format": "json"}]: dispatch > -90> 2015-05-18 06:20:08.949851 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5340 con 0x4609600 > -89> 2015-05-18 06:20:08.949891 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "mon_status", "format": "json"}]=0 v0) v1 -- ?+307 0x4668e00 con 0x460a3c0 > -88> 2015-05-18 06:20:08.949910 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5340 con 0x4609600 > -87> 2015-05-18 06:20:08.949919 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.949920 lease_expire=0.000000 has v0 lc 78630 > -86> 2015-05-18 06:20:08.951317 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 8 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x466a600 con 0x460a3c0 > -85> 2015-05-18 06:20:08.951357 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 > -84> 2015-05-18 06:20:08.951380 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch > -83> 2015-05-18 06:20:08.951382 7f09817a0700 10 log_client _send_to_monlog to self > -82> 2015-05-18 06:20:08.951384 7f09817a0700 10 log_client log_queue is 2 last_log 21 sent 20 num 2 unsent 1 sending 1 > -81> 2015-05-18 06:20:08.951386 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.951382 mon.0 10.8.128.33:6789/0 21 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch > -80> 2015-05-18 06:20:08.951394 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f5580 con 0x4609600 > -79> 2015-05-18 06:20:08.951519 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b400 con 0x460a3c0 > -78> 2015-05-18 06:20:08.951538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f5580 con 0x4609600 > -77> 2015-05-18 06:20:08.951548 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.951548 lease_expire=0.000000 has v0 lc 78630 > -76> 2015-05-18 06:20:08.952793 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 9 ==== mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 ==== 94+0+0 (1843246703 0 0) 0x466b000 con 0x460a3c0 > -75> 2015-05-18 06:20:08.952829 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "health", "detail": "", "format": "json"} v 0) v1 > -74> 2015-05-18 06:20:08.952852 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch > -73> 2015-05-18 06:20:08.952855 7f09817a0700 10 log_client _send_to_monlog to self > -72> 2015-05-18 06:20:08.952856 7f09817a0700 10 log_client log_queue is 3 last_log 22 sent 21 num 3 unsent 1 sending 1 > -71> 2015-05-18 06:20:08.952858 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.952854 mon.0 10.8.128.33:6789/0 22 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"prefix": "health", "detail": "", "format": "json"}]: dispatch > -70> 2015-05-18 06:20:08.952864 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2f40 con 0x4609600 > -69> 2015-05-18 06:20:08.952941 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"prefix": "health", "detail": "", "format": "json"}]=0 v0) v1 -- ?+401 0x466a600 con 0x460a3c0 > -68> 2015-05-18 06:20:08.952960 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2f40 con 0x4609600 > -67> 2015-05-18 06:20:08.952970 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.952970 lease_expire=0.000000 has v0 lc 78630 > -66> 2015-05-18 06:20:08.954184 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184097 10.8.128.33:0/1004034 10 ==== mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 ==== 112+0+0 (3334637970 0 0) 0x4669800 con 0x460a3c0 > -65> 2015-05-18 06:20:08.954227 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"} v 0) v1 > -64> 2015-05-18 06:20:08.954251 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch > -63> 2015-05-18 06:20:08.954255 7f09817a0700 10 log_client _send_to_monlog to self > -62> 2015-05-18 06:20:08.954256 7f09817a0700 10 log_client log_queue is 4 last_log 23 sent 22 num 4 unsent 1 sending 1 > -61> 2015-05-18 06:20:08.954258 7f09817a0700 10 log_client will send 2015-05-18 06:20:08.954254 mon.0 10.8.128.33:6789/0 23 : audit [DBG] from='client.? 10.8.128.33:0/1004034' entity='client.admin' cmd=[{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]: dispatch > -60> 2015-05-18 06:20:08.954266 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f60c0 con 0x4609600 > -59> 2015-05-18 06:20:08.954285 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954286 lease_expire=0.000000 has v0 lc 78630 > -58> 2015-05-18 06:20:08.954605 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004034 -- mon_command_ack([{"dumpcontents": ["pgs_brief"], "prefix": "pg dump", "format": "json"}]=0 dumped pgs_brief in format json v568) v1 -- ?+6616 0x466b000 con 0x460a3c0 > -57> 2015-05-18 06:20:08.954626 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f60c0 con 0x4609600 > -56> 2015-05-18 06:20:08.954636 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos writing c 78062..78630) is_readable = 1 - now=2015-05-18 06:20:08.954636 lease_expire=0.000000 has v0 lc 78630 > -55> 2015-05-18 06:20:08.957218 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'get_command_descriptions' '' to 0x45c0150 returned 2098 bytes > -54> 2015-05-18 06:20:08.965521 7f0984a8b700 1 do_command 'config show' 'format:json > -53> 2015-05-18 06:20:08.970043 7f0984a8b700 1 do_command 'config show' 'format:json result is 26602 bytes > -52> 2015-05-18 06:20:08.970050 7f0984a8b700 5 asok(0x4650000) AdminSocket: request 'config show' '' to 0x45c0050 returned 26602 bytes > -51> 2015-05-18 06:20:09.029915 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).reader couldn't read tag, (0) Success > -50> 2015-05-18 06:20:09.029958 7f097f69b700 2 -- 10.8.128.33:6789/0 >> 10.8.128.33:0/1004034 pipe(0x48e2000 sd=21 :6789 s=2 pgs=1 cs=1 l=1 c=0x460a3c0).fault (0) Success > -49> 2015-05-18 06:20:09.134600 7f097f69b700 1 -- 10.8.128.33:6789/0 >> :/0 pipe(0x48e2000 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x460aaa0).accept sd=21 10.8.128.33:60119/0 > -48> 2015-05-18 06:20:09.134790 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 1 ==== auth(proto 0 30 bytes epoch 0) v1 ==== 60+0+0 (673663173 0 0) 0x47f5a00 con 0x460aaa0 > -47> 2015-05-18 06:20:09.134816 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.134817 lease_expire=0.000000 has v0 lc 78631 > -46> 2015-05-18 06:20:09.134841 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0 > -45> 2015-05-18 06:20:09.134870 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0 > -44> 2015-05-18 06:20:09.135204 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (2513201549 0 0) 0x47f7980 con 0x460aaa0 > -43> 2015-05-18 06:20:09.135222 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135223 lease_expire=0.000000 has v0 lc 78631 > -42> 2015-05-18 06:20:09.135370 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f5a00 con 0x460aaa0 > -41> 2015-05-18 06:20:09.135590 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (692588905 0 0) 0x47f7740 con 0x460aaa0 > -40> 2015-05-18 06:20:09.135605 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.135606 lease_expire=0.000000 has v0 lc 78631 > -39> 2015-05-18 06:20:09.135738 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x47f7980 con 0x460aaa0 > -38> 2015-05-18 06:20:09.135936 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.? 10.8.128.33:0/1004078 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x466a600 con 0x460aaa0 > -37> 2015-05-18 06:20:09.135967 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_map magic: 0 v1 -- ?+0 0x47f7740 con 0x460aaa0 > -36> 2015-05-18 06:20:09.135984 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b85a0 con 0x460aaa0 > -35> 2015-05-18 06:20:09.136000 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 5 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x4668e00 con 0x460aaa0 > -34> 2015-05-18 06:20:09.136101 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f2ac0 con 0x460aaa0 > -33> 2015-05-18 06:20:09.136120 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b90e0 con 0x460aaa0 > -32> 2015-05-18 06:20:09.136139 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 6 ==== mon_subscribe({monmap=2+,osdmap=0}) v2 ==== 42+0+0 (2069260600 0 0) 0x466b000 con 0x460aaa0 > -31> 2015-05-18 06:20:09.136183 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- osd_map(85..85 src has 1..85) v3 -- ?+0 0x47f5e80 con 0x460aaa0 > -30> 2015-05-18 06:20:09.136195 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_subscribe_ack(300s) v1 -- ?+0 0x47b92c0 con 0x460aaa0 > -29> 2015-05-18 06:20:09.143538 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 7 ==== mon_command({"prefix": "status", "format": "json"} v 0) v1 ==== 80+0+0 (1009655278 0 0) 0x4669800 con 0x460aaa0 > -28> 2015-05-18 06:20:09.143603 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "status", "format": "json"} v 0) v1 > -27> 2015-05-18 06:20:09.143635 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch > -26> 2015-05-18 06:20:09.143638 7f09817a0700 10 log_client _send_to_monlog to self > -25> 2015-05-18 06:20:09.143639 7f09817a0700 10 log_client log_queue is 5 last_log 24 sent 23 num 5 unsent 1 sending 1 > -24> 2015-05-18 06:20:09.143642 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.143637 mon.0 10.8.128.33:6789/0 24 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch > -23> 2015-05-18 06:20:09.143652 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f2400 con 0x4609600 > -22> 2015-05-18 06:20:09.143795 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 -- ?+1108 0x466b000 con 0x460aaa0 > -21> 2015-05-18 06:20:09.143817 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f2400 con 0x4609600 > -20> 2015-05-18 06:20:09.143827 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.143828 lease_expire=0.000000 has v0 lc 78631 > -19> 2015-05-18 06:20:09.145454 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 8 ==== mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 ==== 82+0+0 (3917390200 0 0) 0x466a000 con 0x460aaa0 > -18> 2015-05-18 06:20:09.145494 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 > -17> 2015-05-18 06:20:09.145520 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch > -16> 2015-05-18 06:20:09.145522 7f09817a0700 10 log_client _send_to_monlog to self > -15> 2015-05-18 06:20:09.145523 7f09817a0700 10 log_client log_queue is 6 last_log 25 sent 24 num 6 unsent 1 sending 1 > -14> 2015-05-18 06:20:09.145527 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.145522 mon.0 10.8.128.33:6789/0 25 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch > -13> 2015-05-18 06:20:09.145535 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f6540 con 0x4609600 > -12> 2015-05-18 06:20:09.145546 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145547 lease_expire=0.000000 has v0 lc 78631 > -11> 2015-05-18 06:20:09.145779 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:0/1004078 -- mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 -- ?+3436 0x4669800 con 0x460aaa0 > -10> 2015-05-18 06:20:09.145805 7f09817a0700 1 -- 10.8.128.33:6789/0 <== mon.0 10.8.128.33:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x47f6540 con 0x4609600 > -9> 2015-05-18 06:20:09.145814 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.145815 lease_expire=0.000000 has v0 lc 78631 > -8> 2015-05-18 06:20:09.147249 7f09817a0700 1 -- 10.8.128.33:6789/0 <== client.184098 10.8.128.33:0/1004078 9 ==== mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 ==== 95+0+0 (942654298 0 0) 0x466b800 con 0x460aaa0 > -7> 2015-05-18 06:20:09.147309 7f09817a0700 0 mon.Mon@0(leader) e1 handle_command mon_command({"prefix": "osd tree", "epoch": 85, "format": "json"} v 0) v1 > -6> 2015-05-18 06:20:09.147337 7f09817a0700 0 log_channel(audit) log [DBG] : from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch > -5> 2015-05-18 06:20:09.147343 7f09817a0700 10 log_client _send_to_monlog to self > -4> 2015-05-18 06:20:09.147345 7f09817a0700 10 log_client log_queue is 7 last_log 26 sent 25 num 7 unsent 1 sending 1 > -3> 2015-05-18 06:20:09.147347 7f09817a0700 10 log_client will send 2015-05-18 06:20:09.147343 mon.0 10.8.128.33:6789/0 26 : audit [DBG] from='client.? 10.8.128.33:0/1004078' entity='client.admin' cmd=[{"prefix": "osd tree", "epoch": 85, "format": "json"}]: dispatch > -2> 2015-05-18 06:20:09.147355 7f09817a0700 1 -- 10.8.128.33:6789/0 --> 10.8.128.33:6789/0 -- log(1 entries) v1 -- ?+0 0x47f3cc0 con 0x4609600 > -1> 2015-05-18 06:20:09.147365 7f09817a0700 5 mon.Mon@0(leader).paxos(paxos active c 78062..78631) is_readable = 1 - now=2015-05-18 06:20:09.147366 lease_expire=0.000000 has v0 lc 78631 > 0> 2015-05-18 06:20:09.149437 7f09817a0700 -1 *** Caught signal (Aborted) ** > in thread 7f09817a0700 > > ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) > 1: /usr/bin/ceph-mon() [0x9017e2] > 2: (()+0xf130) [0x7f098826e130] > 3: (gsignal()+0x37) [0x7f0986c885d7] > 4: (abort()+0x148) [0x7f0986c89cc8] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f098758c9b5] > 6: (()+0x5e926) [0x7f098758a926] > 7: (()+0x5e953) [0x7f098758a953] > 8: (()+0x5eb73) [0x7f098758ab73] > 9: (std::__throw_logic_error(char const*)+0x77) [0x7f09875df717] > 10: (char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)+0xa1) [0x7f09875eb561] > 11: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0x38) [0x7f09875eb918] > 12: (CrushTreeDumper::dump_item_fields(CrushWrapper const*, CrushTreeDumper::Item const&, ceph::Formatter*)+0xb9) [0x6175c9] > 13: (OSDMap::print_tree(std::ostream*, ceph::Formatter*) const+0x10d9) [0x787ab9] > 14: (OSDMonitor::preprocess_command(MMonCommand*)+0xe55) [0x60a0b5] > 15: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x20b) [0x60f75b] > 16: (PaxosService::dispatch(PaxosServiceMessage*)+0x833) [0x5cacd3] > 17: (Monitor::handle_command(MMonCommand*)+0x147c) [0x591a4c] > 18: (Monitor::dispatch(MonSession*, Message*, bool)+0xf9) [0x594c89] > 19: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x595936] > 20: (Monitor::ms_dispatch(Message*)+0x23) [0x5b5403] > 21: (DispatchQueue::entry()+0x64a) [0x8a1d9a] > 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x79bd9d] > 23: (()+0x7df5) [0x7f0988266df5] > 24: (clone()+0x6d) [0x7f0986d491ad] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-mon.Mon.log >--- end dump of recent events --- > >=================================== > >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1222509
: 1026671 |
1029386
|
1029400
|
1029402