Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 593132 Details for
Bug 820821
Nodes do not agree on CPG membership, messages lost
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Blackbox contents from pcmk-1 after "unknown node abort"
cpg.blackbox (text/plain), 69.16 KB, created by
Andrew Beekhof
on 2012-06-20 05:42:58 UTC
(
hide
)
Description:
Blackbox contents from pcmk-1 after "unknown node abort"
Filename:
MIME Type:
Creator:
Andrew Beekhof
Created:
2012-06-20 05:42:58 UTC
Size:
69.16 KB
patch
obsolete
>[root@pcmk-1 ~]# qb-blackbox /var/lib/corosync/fdata-2012-06-20T12:31:17-18010 >Dumping the contents of /var/lib/corosync/fdata-2012-06-20T12:31:17-18010 >[debug] shm size:8388608; real_size:8388608; rb->word_size:2097152 >[debug] read total of: 8388620 >Ringbuffer: > ->NORMAL > ->write_pt [19003] > ->read_pt [0] > ->size [2097152 words] > =>free [8312592 bytes] > =>used [76012 bytes] >notice Jun 20 12:31:15 main(1068):10: Corosync Cluster Engine ('2.0.1.21-a96650-xdirty'): started and ready to provide service. >info Jun 20 12:31:15 main(1069):10: Corosync built-in features: >debug Jun 20 12:31:15 totemsrp_initialize(837):14: Token Timeout (5000 ms) retransmit timeout (490 ms) >debug Jun 20 12:31:15 totemsrp_initialize(840):14: token hold (382 ms) retransmits before loss (10 retrans) >debug Jun 20 12:31:15 totemsrp_initialize(847):14: join (1000 ms) send_join (45 ms) consensus (7500 ms) merge (200 ms) >debug Jun 20 12:31:15 totemsrp_initialize(850):14: downcheck (1000 ms) fail to recv const (2500 msgs) >debug Jun 20 12:31:15 totemsrp_initialize(852):14: seqno unchanged const (30 rotations) Maximum network MTU 1401 >debug Jun 20 12:31:15 totemsrp_initialize(856):14: window size per rotation (50 messages) maximum messages per rotation (20 messages) >debug Jun 20 12:31:15 totemsrp_initialize(860):14: missed count const (5 messages) >debug Jun 20 12:31:15 totemsrp_initialize(863):14: send threads (0 threads) >debug Jun 20 12:31:15 totemsrp_initialize(866):14: RRP token expired timeout (490 ms) >debug Jun 20 12:31:15 totemsrp_initialize(869):14: RRP token problem counter (2000 ms) >debug Jun 20 12:31:15 totemsrp_initialize(872):14: RRP threshold (10 problem count) >debug Jun 20 12:31:15 totemsrp_initialize(875):14: RRP multicast threshold (100 problem count) >debug Jun 20 12:31:15 totemsrp_initialize(878):14: RRP automatic recovery check timeout (1000 ms) >debug Jun 20 12:31:15 totemsrp_initialize(880):14: RRP mode set to none. >debug Jun 20 12:31:15 totemsrp_initialize(883):14: heartbeat_failures_allowed (0) >debug Jun 20 12:31:15 totemsrp_initialize(885):14: max_network_delay (50 ms) >debug Jun 20 12:31:15 totemsrp_initialize(908):14: HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 >notice Jun 20 12:31:15 totemnet_instance_initialize(241):14: Initializing transport (UDP/IP Multicast). >notice Jun 20 12:31:15 init_nss(530):14: Initializing transmit/receive security (NSS) crypto: none hash: none >debug Jun 20 12:31:16 totemudp_build_sockets_ip(792):14: Receive multicast socket recv buffer size (320000 bytes). >debug Jun 20 12:31:16 totemudp_build_sockets_ip(798):14: Transmit multicast socket send buffer size (320000 bytes). >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 2 for FD 9 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 3 for FD 11 >notice Jun 20 12:31:16 timer_function_netif_check_timeout(602):14: The network interface [192.168.122.101] is now up. >debug Jun 20 12:31:16 main_iface_change_fn(4504):14: Created or loaded sequence id 1090c.192.168.122.101 for this ring. >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync configuration map access [0] >debug Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cmap [0] >info Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cmap >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 4 for FD 12 >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync configuration service [1] >debug Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cfg [1] >info Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cfg >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 5 for FD 13 >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync cluster closed process group service v1.01 [2] >debug Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cpg [2] >info Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cpg >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 6 for FD 14 >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync profile loading service [4] >debug Jun 20 12:31:16 cs_ipcs_service_init(812):10: NOT Initializing IPC on pload [4] >notice Jun 20 12:31:16 quorum_exec_init_fn(277):1: Using quorum provider corosync_votequorum >trace Jun 20 12:31:16 votequorum_init(1857):1: ENTERING votequorum_init() >trace Jun 20 12:31:16 votequorum_exec_init_fn(1718):1: ENTERING votequorum_exec_init_fn() >trace Jun 20 12:31:16 allocate_node(440):1: ENTERING allocate_node() >trace Jun 20 12:31:16 allocate_node(469):1: LEAVING allocate_node() >trace Jun 20 12:31:16 allocate_node(440):1: ENTERING allocate_node() >trace Jun 20 12:31:16 node_add_ordered(414):1: ENTERING node_add_ordered() >trace Jun 20 12:31:16 node_add_ordered(432):1: LEAVING node_add_ordered() >trace Jun 20 12:31:16 allocate_node(469):1: LEAVING allocate_node() >trace Jun 20 12:31:16 votequorum_readconfig(895):1: ENTERING votequorum_readconfig() >debug Jun 20 12:31:16 votequorum_readconfig(897):1: Reading configuration (runtime: 0) >trace Jun 20 12:31:16 votequorum_read_nodelist_configuration(815):1: ENTERING votequorum_read_nodelist_configuration() >debug Jun 20 12:31:16 votequorum_read_nodelist_configuration(819):1: No nodelist defined or our node is not in the nodelist >trace Jun 20 12:31:16 votequorum_qdevice_is_configured(864):1: ENTERING votequorum_qdevice_is_configured() >trace Jun 20 12:31:16 votequorum_qdevice_is_configured(879):1: LEAVING votequorum_qdevice_is_configured() >trace Jun 20 12:31:16 votequorum_readconfig(1109):1: LEAVING votequorum_readconfig() >trace Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4 >trace Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >trace Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:16 votequorum_exec_add_config_notification(1141):1: ENTERING votequorum_exec_add_config_notification() >trace Jun 20 12:31:16 votequorum_exec_add_config_notification(1155):1: LEAVING votequorum_exec_add_config_notification() >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >debug Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:16 votequorum_exec_init_fn(1766):1: LEAVING votequorum_exec_init_fn() >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync vote quorum service v1.0 [5] >debug Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on votequorum [5] >info Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: votequorum >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 7 for FD 15 >trace Jun 20 12:31:16 votequorum_init(1872):1: LEAVING votequorum_init() >notice Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync cluster quorum service v0.1 [3] >debug Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on quorum [3] >info Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: quorum >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 8 for FD 16 >debug Jun 20 12:31:16 memb_state_gather_enter(1979):14: entering GATHER state from 15. >debug Jun 20 12:31:16 memb_state_commit_token_create(3020):14: Creating commit token because I am the rep. >debug Jun 20 12:31:16 old_ring_state_save(1471):14: Saving state aru 0 high seq received 0 >debug Jun 20 12:31:16 memb_ring_id_set_and_store(3265):14: Storing new sequence id for ring 10910 >debug Jun 20 12:31:16 memb_state_commit_enter(2035):14: entering COMMIT state. >debug Jun 20 12:31:16 message_handler_memb_commit_token(4351):14: got commit token >debug Jun 20 12:31:16 memb_state_recovery_enter(2072):14: entering RECOVERY state. >debug Jun 20 12:31:16 memb_state_recovery_enter(2118):14: position [0] member 192.168.122.101: >debug Jun 20 12:31:16 memb_state_recovery_enter(2122):14: previous ring seq 1090c rep 192.168.122.101 >debug Jun 20 12:31:16 memb_state_recovery_enter(2128):14: aru 0 high delivered 0 received flag 1 >debug Jun 20 12:31:16 memb_state_recovery_enter(2226):14: Did not need to originate any messages in recovery. >debug Jun 20 12:31:16 message_handler_memb_commit_token(4351):14: got commit token >debug Jun 20 12:31:16 message_handler_memb_commit_token(4404):14: Sending initial ORF token >debug Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0 >debug Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 >debug Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 >debug Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 >debug Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:16 message_handler_orf_token(3696):14: retrans flag count 4 token aru 0 install seq 0 aru 0 0 >debug Jun 20 12:31:16 old_ring_state_reset(1487):14: Resetting old ring state >debug Jun 20 12:31:16 deliver_messages_from_recovery_to_regular(1693):14: recovery to regular 1-0 >debug Jun 20 12:31:16 memb_state_operational_enter(1779):14: Delivering to app 1 to 0 >trace Jun 20 12:31:16 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn() >trace Jun 20 12:31:16 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn() >trace Jun 20 12:31:16 sync_abort(594):2: ENTERING sync_abort() >trace Jun 20 12:31:16 sync_save_transitional(586):2: ENTERING sync_save_transitional() >debug Jun 20 12:31:16 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.101) >trace Jun 20 12:31:16 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn() >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >debug Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >notice Jun 20 12:31:16 log_view_list(132):1: Members[1]: 101 >debug Jun 20 12:31:16 send_library_notification(362):1: sending quorum notification to (nil), length = 52 >trace Jun 20 12:31:16 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn() >trace Jun 20 12:31:16 sync_abort(594):2: ENTERING sync_abort() >trace Jun 20 12:31:16 sync_start(568):2: ENTERING sync_start() >debug Jun 20 12:31:16 memb_state_operational_enter(1905):14: entering OPERATIONAL state. >notice Jun 20 12:31:16 memb_state_operational_enter(1909):14: A processor joined or left the membership and a new membership (192.168.122.101:67856) was formed. >debug Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:16 messages_deliver_to_app(3768):14: Delivering 0 to 2 >debug Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1 to pending delivery queue >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8 >debug Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4 >trace Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >trace Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 2 to pending delivery queue >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8 >debug Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4 >trace Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >trace Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:16 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4 >trace Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >trace Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:16 messages_deliver_to_app(3768):14: Delivering 2 to 4 >debug Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 3 to pending delivery queue >debug Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 4 to pending delivery queue >debug Jun 20 12:31:16 downlist_log(772):5: comparing: sender r(0) ip(192.168.122.101) ; members(old:0 left:0) >debug Jun 20 12:31:16 downlist_log(772):5: chosen downlist: sender r(0) ip(192.168.122.101) ; members(old:0 left:0) >debug Jun 20 12:31:16 sync_barrier_handler(250):2: Committing synchronization for corosync cluster closed process group service v1.01 >notice Jun 20 12:31:16 corosync_sync_completed(273):10: Completed service synchronization, ready to provide service. >debug Jun 20 12:31:16 messages_free(2427):14: releasing messages up to and including 2 >debug Jun 20 12:31:16 messages_free(2427):14: releasing messages up to and including 4 >debug Jun 20 12:31:16 memb_state_gather_enter(1979):14: entering GATHER state from 11. >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18017-17) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18017] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 9 for FD 17 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >trace Jun 20 12:31:16 cfg_lib_init_fn(453):3: ENTERING cfg_lib_init_fn() >trace Jun 20 12:31:16 cfg_lib_init_fn(455):3: LEAVING cfg_lib_init_fn() >trace Jun 20 12:31:16 message_handler_req_lib_cfg_ringstatusget(547):3: ENTERING message_handler_req_lib_cfg_ringstatusget() >trace Jun 20 12:31:16 message_handler_req_lib_cfg_ringstatusget(575):3: LEAVING message_handler_req_lib_cfg_ringstatusget() >debug Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18017-17) >debug Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18017-17) state:2 >trace Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect() >debug Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9) >debug Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() >trace Jun 20 12:31:16 cfg_lib_exit_fn(443):3: ENTERING cfg_lib_exit_fn() >trace Jun 20 12:31:16 remove_ci_from_shutdown(402):3: ENTERING remove_ci_from_shutdown() >trace Jun 20 12:31:16 remove_ci_from_shutdown(435):3: LEAVING remove_ci_from_shutdown() >trace Jun 20 12:31:16 cfg_lib_exit_fn(445):3: LEAVING cfg_lib_exit_fn() >debug Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-response-18010-18017-17-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-event-18010-18017-17-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-request-18010-18017-17-header >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-17) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cmap_lib_init_fn(185):11: lib_init_fn: conn=0xbc7aa0 >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-18) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 10 for FD 18 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cmap_lib_init_fn(185):11: lib_init_fn: conn=0xbc6b50 >debug Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18031-18) >debug Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18031-18) state:2 >trace Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect() >debug Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9) >debug Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() >debug Jun 20 12:31:16 cmap_lib_exit_fn(204):11: exit_fn for conn=0xbc6b50 >debug Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-response-18010-18031-18-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-event-18010-18031-18-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-request-18010-18031-18-header >debug Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18031-17) >debug Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18031-17) state:2 >trace Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect() >debug Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9) >debug Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() >debug Jun 20 12:31:16 cmap_lib_exit_fn(204):11: exit_fn for conn=0xbc7aa0 >debug Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-response-18010-18031-17-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-event-18010-18031-17-header >debug Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-request-18010-18031-17-header >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-17) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >trace Jun 20 12:31:16 cfg_lib_init_fn(453):3: ENTERING cfg_lib_init_fn() >trace Jun 20 12:31:16 cfg_lib_init_fn(455):3: LEAVING cfg_lib_init_fn() >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-18) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbc3db0, cpd=0xbc4354 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0 >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18036-19) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18036] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 11 for FD 19 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbc6f60, cpd=0xbc7584 >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18038-20) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18038] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 12 for FD 20 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbcb5e0, cpd=0xbcbb44 >debug Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18034-21) >debug Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18034] >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >debug Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576 >trace Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 13 for FD 21 >debug Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created >debug Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbccf40, cpd=0xbcd524 >debug Jun 20 12:31:17 memb_state_commit_token_create(3020):14: Creating commit token because I am the rep. >debug Jun 20 12:31:17 old_ring_state_save(1471):14: Saving state aru 4 high seq received 4 >debug Jun 20 12:31:17 memb_ring_id_set_and_store(3265):14: Storing new sequence id for ring 10920 >debug Jun 20 12:31:17 memb_state_commit_enter(2035):14: entering COMMIT state. >debug Jun 20 12:31:17 message_handler_memb_commit_token(4351):14: got commit token >debug Jun 20 12:31:17 memb_state_recovery_enter(2072):14: entering RECOVERY state. >debug Jun 20 12:31:17 memb_state_recovery_enter(2114):14: TRANS [0] member 192.168.122.101: >debug Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [0] member 192.168.122.101: >debug Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 10910 rep 192.168.122.101 >debug Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 4 high delivered 4 received flag 1 >debug Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [1] member 192.168.122.102: >debug Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102 >debug Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1 >debug Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [2] member 192.168.122.103: >debug Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102 >debug Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1 >debug Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [3] member 192.168.122.104: >debug Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102 >debug Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1 >debug Jun 20 12:31:17 memb_state_recovery_enter(2226):14: Did not need to originate any messages in recovery. >debug Jun 20 12:31:17 message_handler_memb_commit_token(4351):14: got commit token >debug Jun 20 12:31:17 message_handler_memb_commit_token(4404):14: Sending initial ORF token >debug Jun 20 12:31:17 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0 >debug Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 >debug Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 >debug Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 >debug Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0 >debug Jun 20 12:31:17 message_handler_orf_token(3696):14: retrans flag count 4 token aru 0 install seq 0 aru 0 0 >debug Jun 20 12:31:17 old_ring_state_reset(1487):14: Resetting old ring state >debug Jun 20 12:31:17 deliver_messages_from_recovery_to_regular(1693):14: recovery to regular 1-0 >debug Jun 20 12:31:17 memb_state_operational_enter(1779):14: Delivering to app 5 to 4 >trace Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2 >trace Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2 >trace Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2 >trace Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2 >trace Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2 >trace Jun 20 12:31:17 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn() >trace Jun 20 12:31:17 sync_save_transitional(586):2: ENTERING sync_save_transitional() >debug Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.102) >debug Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.103) >debug Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.104) >trace Jun 20 12:31:17 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo() >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo() >notice Jun 20 12:31:17 log_view_list(132):1: Members[4]: 101 102 103 104 >debug Jun 20 12:31:17 send_library_notification(362):1: sending quorum notification to (nil), length = 64 >trace Jun 20 12:31:17 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn() >trace Jun 20 12:31:17 sync_abort(594):2: ENTERING sync_abort() >trace Jun 20 12:31:17 sync_start(568):2: ENTERING sync_start() >debug Jun 20 12:31:17 memb_state_operational_enter(1905):14: entering OPERATIONAL state. >notice Jun 20 12:31:17 memb_state_operational_enter(1909):14: A processor joined or left the membership and a new membership (192.168.122.101:67872) was formed. >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 0 to 1 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 2 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1 to 2 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 2 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 3 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 2 to 3 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 3 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 4 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 3 to 4 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 4 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 5 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 4 to 5 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 5 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 6 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 5 to 6 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 6 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 7 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 6 to 7 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 7 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 8 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 7 to 8 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 8 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 9 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 8 to 9 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 9 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq a >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 9 to a >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq a to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq b >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering a to b >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq b to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq c >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering b to c >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq c to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq d >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering c to d >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq d to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq e >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering d to e >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq e to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq f >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering e to f >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq f to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 10 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering f to 10 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 10 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 11 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 10 to 11 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 11 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 12 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 11 to 12 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 12 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 13 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 12 to 13 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 13 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 14 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 13 to 14 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 14 to pending delivery queue >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 15 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 14 to 15 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 15 to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103 >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 16 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 15 to 16 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 16 to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103 >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 17 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 16 to 17 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 17 to pending delivery queue >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[103]: votes: 1, expected: 4 flags: 1 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(499):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 allocate_node(440):1: ENTERING allocate_node() >trace Jun 20 12:31:17 node_add_ordered(414):1: ENTERING node_add_ordered() >trace Jun 20 12:31:17 node_add_ordered(432):1: LEAVING node_add_ordered() >trace Jun 20 12:31:17 allocate_node(469):1: LEAVING allocate_node() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[103]: votes: 1, expected: 4 flags: 1 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(494):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 18 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 17 to 18 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 18 to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104 >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 19 >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 18 to 19 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 19 to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104 >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1a >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 19 to 1a >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1a to pending delivery queue >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[104]: votes: 1, expected: 4 flags: 1 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(499):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 allocate_node(440):1: ENTERING allocate_node() >trace Jun 20 12:31:17 node_add_ordered(414):1: ENTERING node_add_ordered() >trace Jun 20 12:31:17 node_add_ordered(432):1: LEAVING node_add_ordered() >trace Jun 20 12:31:17 allocate_node(469):1: LEAVING allocate_node() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >debug Jun 20 12:31:17 are_we_quorate(712):1: quorum regained, resuming activity >notice Jun 20 12:31:17 quorum_api_set_quorum(148):1: This node is within the primary component and will provide service. >notice Jun 20 12:31:17 log_view_list(132):1: Members[4]: 101 102 103 104 >debug Jun 20 12:31:17 send_library_notification(362):1: sending quorum notification to (nil), length = 64 >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[104]: votes: 1, expected: 4 flags: 1 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(494):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 totemsrp_mcast(2304):14: mcasted message added to pending queue >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1a to 1d >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1b to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1c to pending delivery queue >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101 >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1d to pending delivery queue >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 0 >debug Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101 >debug Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0 >trace Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid() >trace Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid() >trace Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum() >trace Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes() >trace Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes() >debug Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4 >trace Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum() >debug Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4 >debug Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4 >trace Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum() >trace Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate() >trace Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id() >debug Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101 >trace Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id() >trace Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification() >trace Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum() >trace Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo() >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1b >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1c >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1d >debug Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1e >debug Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1d to 1e >debug Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1e to pending delivery queue >warning Jun 20 12:31:17 message_handler_req_exec_cpg_mcast(1370):5: Unknown node 66 -> we will not deliver message >[debug] Free'ing ringbuffer: /dev/shm/qb-create_from_file-header >[root@pcmk-1 ~]#
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 820821
:
583724
|
584502
|
584503
|
584918
|
584944
|
589886
|
590918
|
591194
|
591509
|
591532
|
592063
| 593132 |
593133
|
640873
|
640874
|
640875