Attachment 593132 Details for Bug 820821 – Blackbox contents from pcmk-1 after "unknown node abort"

Blackbox contents from pcmk-1 after "unknown node abort"

cpg.blackbox (text/plain), 69.16 KB, created by Andrew Beekhof on 2012-06-20 05:42:58 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Andrew Beekhof

Created: 2012-06-20 05:42:58 UTC

Size: 69.16 KB

patch

obsolete

>[root@pcmk-1 ~]# qb-blackbox /var/lib/corosync/fdata-2012-06-20T12:31:17-18010
>Dumping the contents of /var/lib/corosync/fdata-2012-06-20T12:31:17-18010
>[debug] shm size:8388608; real_size:8388608; rb->word_size:2097152
>[debug] read total of: 8388620
>Ringbuffer: 
> ->NORMAL
> ->write_pt [19003]
> ->read_pt [0]
> ->size [2097152 words]
> =>free [8312592 bytes]
> =>used [76012 bytes]
>notice  Jun 20 12:31:15 main(1068):10: Corosync Cluster Engine ('2.0.1.21-a96650-xdirty'): started and ready to provide service.
>info    Jun 20 12:31:15 main(1069):10: Corosync built-in features:
>debug   Jun 20 12:31:15 totemsrp_initialize(837):14: Token Timeout (5000 ms) retransmit timeout (490 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(840):14: token hold (382 ms) retransmits before loss (10 retrans)
>debug   Jun 20 12:31:15 totemsrp_initialize(847):14: join (1000 ms) send_join (45 ms) consensus (7500 ms) merge (200 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(850):14: downcheck (1000 ms) fail to recv const (2500 msgs)
>debug   Jun 20 12:31:15 totemsrp_initialize(852):14: seqno unchanged const (30 rotations) Maximum network MTU 1401
>debug   Jun 20 12:31:15 totemsrp_initialize(856):14: window size per rotation (50 messages) maximum messages per rotation (20 messages)
>debug   Jun 20 12:31:15 totemsrp_initialize(860):14: missed count const (5 messages)
>debug   Jun 20 12:31:15 totemsrp_initialize(863):14: send threads (0 threads)
>debug   Jun 20 12:31:15 totemsrp_initialize(866):14: RRP token expired timeout (490 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(869):14: RRP token problem counter (2000 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(872):14: RRP threshold (10 problem count)
>debug   Jun 20 12:31:15 totemsrp_initialize(875):14: RRP multicast threshold (100 problem count)
>debug   Jun 20 12:31:15 totemsrp_initialize(878):14: RRP automatic recovery check timeout (1000 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(880):14: RRP mode set to none.
>debug   Jun 20 12:31:15 totemsrp_initialize(883):14: heartbeat_failures_allowed (0)
>debug   Jun 20 12:31:15 totemsrp_initialize(885):14: max_network_delay (50 ms)
>debug   Jun 20 12:31:15 totemsrp_initialize(908):14: HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
>notice  Jun 20 12:31:15 totemnet_instance_initialize(241):14: Initializing transport (UDP/IP Multicast).
>notice  Jun 20 12:31:15 init_nss(530):14: Initializing transmit/receive security (NSS) crypto: none hash: none
>debug   Jun 20 12:31:16 totemudp_build_sockets_ip(792):14: Receive multicast socket recv buffer size (320000 bytes).
>debug   Jun 20 12:31:16 totemudp_build_sockets_ip(798):14: Transmit multicast socket send buffer size (320000 bytes).
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 2 for FD 9
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 3 for FD 11
>notice  Jun 20 12:31:16 timer_function_netif_check_timeout(602):14: The network interface [192.168.122.101] is now up.
>debug   Jun 20 12:31:16 main_iface_change_fn(4504):14: Created or loaded sequence id 1090c.192.168.122.101 for this ring.
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync configuration map access [0]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cmap [0]
>info    Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cmap
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 4 for FD 12
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync configuration service [1]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cfg [1]
>info    Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cfg
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 5 for FD 13
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync cluster closed process group service v1.01 [2]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on cpg [2]
>info    Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: cpg
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 6 for FD 14
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync profile loading service [4]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(812):10: NOT Initializing IPC on pload [4]
>notice  Jun 20 12:31:16 quorum_exec_init_fn(277):1: Using quorum provider corosync_votequorum
>trace   Jun 20 12:31:16 votequorum_init(1857):1: ENTERING votequorum_init()
>trace   Jun 20 12:31:16 votequorum_exec_init_fn(1718):1: ENTERING votequorum_exec_init_fn()
>trace   Jun 20 12:31:16 allocate_node(440):1: ENTERING allocate_node()
>trace   Jun 20 12:31:16 allocate_node(469):1: LEAVING allocate_node()
>trace   Jun 20 12:31:16 allocate_node(440):1: ENTERING allocate_node()
>trace   Jun 20 12:31:16 node_add_ordered(414):1: ENTERING node_add_ordered()
>trace   Jun 20 12:31:16 node_add_ordered(432):1: LEAVING node_add_ordered()
>trace   Jun 20 12:31:16 allocate_node(469):1: LEAVING allocate_node()
>trace   Jun 20 12:31:16 votequorum_readconfig(895):1: ENTERING votequorum_readconfig()
>debug   Jun 20 12:31:16 votequorum_readconfig(897):1: Reading configuration (runtime: 0)
>trace   Jun 20 12:31:16 votequorum_read_nodelist_configuration(815):1: ENTERING votequorum_read_nodelist_configuration()
>debug   Jun 20 12:31:16 votequorum_read_nodelist_configuration(819):1: No nodelist defined or our node is not in the nodelist
>trace   Jun 20 12:31:16 votequorum_qdevice_is_configured(864):1: ENTERING votequorum_qdevice_is_configured()
>trace   Jun 20 12:31:16 votequorum_qdevice_is_configured(879):1: LEAVING votequorum_qdevice_is_configured()
>trace   Jun 20 12:31:16 votequorum_readconfig(1109):1: LEAVING votequorum_readconfig()
>trace   Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4
>trace   Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>trace   Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:16 votequorum_exec_add_config_notification(1141):1: ENTERING votequorum_exec_add_config_notification()
>trace   Jun 20 12:31:16 votequorum_exec_add_config_notification(1155):1: LEAVING votequorum_exec_add_config_notification()
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>debug   Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:16 votequorum_exec_init_fn(1766):1: LEAVING votequorum_exec_init_fn()
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync vote quorum service v1.0 [5]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on votequorum [5]
>info    Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: votequorum
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 7 for FD 15
>trace   Jun 20 12:31:16 votequorum_init(1872):1: LEAVING votequorum_init()
>notice  Jun 20 12:31:16 corosync_service_link_and_init(176):13: Service engine loaded: corosync cluster quorum service v0.1 [3]
>debug   Jun 20 12:31:16 cs_ipcs_service_init(820):10: Initializing IPC on quorum [3]
>info    Jun 20 12:31:16 qb_ipcs_us_publish(530):11: server name: quorum
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 8 for FD 16
>debug   Jun 20 12:31:16 memb_state_gather_enter(1979):14: entering GATHER state from 15.
>debug   Jun 20 12:31:16 memb_state_commit_token_create(3020):14: Creating commit token because I am the rep.
>debug   Jun 20 12:31:16 old_ring_state_save(1471):14: Saving state aru 0 high seq received 0
>debug   Jun 20 12:31:16 memb_ring_id_set_and_store(3265):14: Storing new sequence id for ring 10910
>debug   Jun 20 12:31:16 memb_state_commit_enter(2035):14: entering COMMIT state.
>debug   Jun 20 12:31:16 message_handler_memb_commit_token(4351):14: got commit token
>debug   Jun 20 12:31:16 memb_state_recovery_enter(2072):14: entering RECOVERY state.
>debug   Jun 20 12:31:16 memb_state_recovery_enter(2118):14: position [0] member 192.168.122.101:
>debug   Jun 20 12:31:16 memb_state_recovery_enter(2122):14: previous ring seq 1090c rep 192.168.122.101
>debug   Jun 20 12:31:16 memb_state_recovery_enter(2128):14: aru 0 high delivered 0 received flag 1
>debug   Jun 20 12:31:16 memb_state_recovery_enter(2226):14: Did not need to originate any messages in recovery.
>debug   Jun 20 12:31:16 message_handler_memb_commit_token(4351):14: got commit token
>debug   Jun 20 12:31:16 message_handler_memb_commit_token(4404):14: Sending initial ORF token
>debug   Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:16 message_handler_orf_token(3696):14: retrans flag count 4 token aru 0 install seq 0 aru 0 0
>debug   Jun 20 12:31:16 old_ring_state_reset(1487):14: Resetting old ring state
>debug   Jun 20 12:31:16 deliver_messages_from_recovery_to_regular(1693):14: recovery to regular 1-0
>debug   Jun 20 12:31:16 memb_state_operational_enter(1779):14: Delivering to app 1 to 0
>trace   Jun 20 12:31:16 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn()
>trace   Jun 20 12:31:16 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn()
>trace   Jun 20 12:31:16 sync_abort(594):2: ENTERING sync_abort()
>trace   Jun 20 12:31:16 sync_save_transitional(586):2: ENTERING sync_save_transitional()
>debug   Jun 20 12:31:16 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.101) 
>trace   Jun 20 12:31:16 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn()
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>debug   Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:16 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>notice  Jun 20 12:31:16 log_view_list(132):1: Members[1]: 101
>debug   Jun 20 12:31:16 send_library_notification(362):1: sending quorum notification to (nil), length = 52
>trace   Jun 20 12:31:16 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn()
>trace   Jun 20 12:31:16 sync_abort(594):2: ENTERING sync_abort()
>trace   Jun 20 12:31:16 sync_start(568):2: ENTERING sync_start()
>debug   Jun 20 12:31:16 memb_state_operational_enter(1905):14: entering OPERATIONAL state.
>notice  Jun 20 12:31:16 memb_state_operational_enter(1909):14: A processor joined or left the membership and a new membership (192.168.122.101:67856) was formed.
>debug   Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:16 messages_deliver_to_app(3768):14: Delivering 0 to 2
>debug   Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1 to pending delivery queue
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8
>debug   Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4
>trace   Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>trace   Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 2 to pending delivery queue
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8
>debug   Jun 20 12:31:16 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4
>trace   Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>trace   Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:16 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:16 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:16 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:16 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:16 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:16 recalculate_quorum(784):1: total_votes=1, expected_votes=4
>trace   Jun 20 12:31:16 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:16 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>trace   Jun 20 12:31:16 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:16 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:16 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:16 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:16 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:16 messages_deliver_to_app(3768):14: Delivering 2 to 4
>debug   Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 3 to pending delivery queue
>debug   Jun 20 12:31:16 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 4 to pending delivery queue
>debug   Jun 20 12:31:16 downlist_log(772):5: comparing: sender r(0) ip(192.168.122.101) ; members(old:0 left:0)
>debug   Jun 20 12:31:16 downlist_log(772):5: chosen downlist: sender r(0) ip(192.168.122.101) ; members(old:0 left:0)
>debug   Jun 20 12:31:16 sync_barrier_handler(250):2: Committing synchronization for corosync cluster closed process group service v1.01
>notice  Jun 20 12:31:16 corosync_sync_completed(273):10: Completed service synchronization, ready to provide service.
>debug   Jun 20 12:31:16 messages_free(2427):14: releasing messages up to and including 2
>debug   Jun 20 12:31:16 messages_free(2427):14: releasing messages up to and including 4
>debug   Jun 20 12:31:16 memb_state_gather_enter(1979):14: entering GATHER state from 11.
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18017-17)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18017]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 9 for FD 17
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>trace   Jun 20 12:31:16 cfg_lib_init_fn(453):3: ENTERING cfg_lib_init_fn()
>trace   Jun 20 12:31:16 cfg_lib_init_fn(455):3: LEAVING cfg_lib_init_fn()
>trace   Jun 20 12:31:16 message_handler_req_lib_cfg_ringstatusget(547):3: ENTERING message_handler_req_lib_cfg_ringstatusget()
>trace   Jun 20 12:31:16 message_handler_req_lib_cfg_ringstatusget(575):3: LEAVING message_handler_req_lib_cfg_ringstatusget()
>debug   Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18017-17)
>debug   Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18017-17) state:2
>trace   Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect()
>debug   Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9)
>debug   Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() 
>trace   Jun 20 12:31:16 cfg_lib_exit_fn(443):3: ENTERING cfg_lib_exit_fn()
>trace   Jun 20 12:31:16 remove_ci_from_shutdown(402):3: ENTERING remove_ci_from_shutdown()
>trace   Jun 20 12:31:16 remove_ci_from_shutdown(435):3: LEAVING remove_ci_from_shutdown()
>trace   Jun 20 12:31:16 cfg_lib_exit_fn(445):3: LEAVING cfg_lib_exit_fn()
>debug   Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() 
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-response-18010-18017-17-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-event-18010-18017-17-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cfg-request-18010-18017-17-header
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-17)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cmap_lib_init_fn(185):11: lib_init_fn: conn=0xbc7aa0
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-18)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 10 for FD 18
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cmap_lib_init_fn(185):11: lib_init_fn: conn=0xbc6b50
>debug   Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18031-18)
>debug   Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18031-18) state:2
>trace   Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect()
>debug   Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9)
>debug   Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() 
>debug   Jun 20 12:31:16 cmap_lib_exit_fn(204):11: exit_fn for conn=0xbc6b50
>debug   Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() 
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-response-18010-18031-18-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-event-18010-18031-18-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-request-18010-18031-18-header
>debug   Jun 20 12:31:16 qb_ipcs_dispatch_connection_request(727):11: HUP conn (18010-18031-17)
>debug   Jun 20 12:31:16 qb_ipcs_disconnect(567):11: qb_ipcs_disconnect(18010-18031-17) state:2
>trace   Jun 20 12:31:16 qb_ipcs_sockets_disconnect(1024):11: ENTERING qb_ipcs_sockets_disconnect()
>debug   Jun 20 12:31:16 _del(117):11: epoll_ctl(del): Bad file descriptor (9)
>debug   Jun 20 12:31:16 cs_ipcs_connection_closed(415):10: cs_ipcs_connection_closed() 
>debug   Jun 20 12:31:16 cmap_lib_exit_fn(204):11: exit_fn for conn=0xbc7aa0
>debug   Jun 20 12:31:16 cs_ipcs_connection_destroyed(388):10: cs_ipcs_connection_destroyed() 
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-response-18010-18031-17-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-event-18010-18031-17-header
>debug   Jun 20 12:31:16 qb_rb_close(272):11: Free'ing ringbuffer: /dev/shm/qb-cmap-request-18010-18031-17-header
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-17)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>trace   Jun 20 12:31:16 cfg_lib_init_fn(453):3: ENTERING cfg_lib_init_fn()
>trace   Jun 20 12:31:16 cfg_lib_init_fn(455):3: LEAVING cfg_lib_init_fn()
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18031-18)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18031]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbc3db0, cpd=0xbc4354
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:16 message_handler_req_lib_cpg_mcast(1808):5: got mcast request on 0xbc3db0
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18036-19)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18036]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 11 for FD 19
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbc6f60, cpd=0xbc7584
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18038-20)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18038]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 12 for FD 20
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbcb5e0, cpd=0xbcbb44
>debug   Jun 20 12:31:16 handle_new_connection(634):11: IPC credentials authenticated (18010-18034-21)
>debug   Jun 20 12:31:16 qb_ipcs_shm_connect(243):11: connecting to client [18034]
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>debug   Jun 20 12:31:16 qb_rb_open(210):11: shm size:zd; real_size:zd; rb->word_size:1048576
>trace   Jun 20 12:31:16 qb_loop_poll_add(357):11: grown poll array to 13 for FD 21
>debug   Jun 20 12:31:16 cs_ipcs_connection_created(270):10: connection created
>debug   Jun 20 12:31:16 cpg_lib_init_fn(1456):5: lib_init_fn: conn=0xbccf40, cpd=0xbcd524
>debug   Jun 20 12:31:17 memb_state_commit_token_create(3020):14: Creating commit token because I am the rep.
>debug   Jun 20 12:31:17 old_ring_state_save(1471):14: Saving state aru 4 high seq received 4
>debug   Jun 20 12:31:17 memb_ring_id_set_and_store(3265):14: Storing new sequence id for ring 10920
>debug   Jun 20 12:31:17 memb_state_commit_enter(2035):14: entering COMMIT state.
>debug   Jun 20 12:31:17 message_handler_memb_commit_token(4351):14: got commit token
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2072):14: entering RECOVERY state.
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2114):14: TRANS [0] member 192.168.122.101:
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [0] member 192.168.122.101:
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 10910 rep 192.168.122.101
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 4 high delivered 4 received flag 1
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [1] member 192.168.122.102:
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [2] member 192.168.122.103:
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2118):14: position [3] member 192.168.122.104:
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2122):14: previous ring seq 1091c rep 192.168.122.102
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2128):14: aru 9 high delivered 9 received flag 1
>debug   Jun 20 12:31:17 memb_state_recovery_enter(2226):14: Did not need to originate any messages in recovery.
>debug   Jun 20 12:31:17 message_handler_memb_commit_token(4351):14: got commit token
>debug   Jun 20 12:31:17 message_handler_memb_commit_token(4404):14: Sending initial ORF token
>debug   Jun 20 12:31:17 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3666):14: token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3677):14: install seq 0 aru 0 high seq received 0
>debug   Jun 20 12:31:17 message_handler_orf_token(3696):14: retrans flag count 4 token aru 0 install seq 0 aru 0 0
>debug   Jun 20 12:31:17 old_ring_state_reset(1487):14: Resetting old ring state
>debug   Jun 20 12:31:17 deliver_messages_from_recovery_to_regular(1693):14: recovery to regular 1-0
>debug   Jun 20 12:31:17 memb_state_operational_enter(1779):14: Delivering to app 5 to 4
>trace   Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2
>trace   Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2
>trace   Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2
>trace   Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2
>trace   Jun 20 12:31:17 qb_ipc_shm_fc_set(128):11: setting fc to 2
>trace   Jun 20 12:31:17 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn()
>trace   Jun 20 12:31:17 sync_save_transitional(586):2: ENTERING sync_save_transitional()
>debug   Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.102) 
>debug   Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.103) 
>debug   Jun 20 12:31:17 member_object_joined(322):10: Member joined: r(0) ip(192.168.122.104) 
>trace   Jun 20 12:31:17 votequorum_confchg_fn(1797):1: ENTERING votequorum_confchg_fn()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1196):1: ENTERING votequorum_exec_send_nodeinfo()
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 votequorum_exec_send_nodeinfo(1219):1: LEAVING votequorum_exec_send_nodeinfo()
>notice  Jun 20 12:31:17 log_view_list(132):1: Members[4]: 101 102 103 104
>debug   Jun 20 12:31:17 send_library_notification(362):1: sending quorum notification to (nil), length = 64
>trace   Jun 20 12:31:17 votequorum_confchg_fn(1848):1: LEAVING votequorum_confchg_fn()
>trace   Jun 20 12:31:17 sync_abort(594):2: ENTERING sync_abort()
>trace   Jun 20 12:31:17 sync_start(568):2: ENTERING sync_start()
>debug   Jun 20 12:31:17 memb_state_operational_enter(1905):14: entering OPERATIONAL state.
>notice  Jun 20 12:31:17 memb_state_operational_enter(1909):14: A processor joined or left the membership and a new membership (192.168.122.101:67872) was formed.
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 0 to 1
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 2
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1 to 2
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 2 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 3
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 2 to 3
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 3 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 4
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 3 to 4
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 4 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 5
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 4 to 5
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 5 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 6
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 5 to 6
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 6 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 7
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 6 to 7
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 7 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 8
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 7 to 8
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 8 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 9
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 8 to 9
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 9 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq a
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 9 to a
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq a to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq b
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering a to b
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq b to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq c
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering b to c
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq c to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq d
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering c to d
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq d to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq e
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering d to e
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq e to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq f
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering e to f
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq f to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 10
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering f to 10
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 10 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 11
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 10 to 11
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 11 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 12
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 11 to 12
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 12 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 13
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 12 to 13
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 13 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 14
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 13 to 14
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 14 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 15
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 14 to 15
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 15 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 16
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 15 to 16
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 16 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 17
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 16 to 17
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 17 to pending delivery queue
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[103]: votes: 1, expected: 4 flags: 1
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(499):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 allocate_node(440):1: ENTERING allocate_node()
>trace   Jun 20 12:31:17 node_add_ordered(414):1: ENTERING node_add_ordered()
>trace   Jun 20 12:31:17 node_add_ordered(432):1: LEAVING node_add_ordered()
>trace   Jun 20 12:31:17 allocate_node(469):1: LEAVING allocate_node()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[103]: votes: 1, expected: 4 flags: 1
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(494):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 103
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=2, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 18
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 17 to 18
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 18 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 19
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 18 to 19
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 19 to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1a
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 19 to 1a
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1a to pending delivery queue
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[104]: votes: 1, expected: 4 flags: 1
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(499):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 allocate_node(440):1: ENTERING allocate_node()
>trace   Jun 20 12:31:17 node_add_ordered(414):1: ENTERING node_add_ordered()
>trace   Jun 20 12:31:17 node_add_ordered(432):1: LEAVING node_add_ordered()
>trace   Jun 20 12:31:17 allocate_node(469):1: LEAVING allocate_node()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>debug   Jun 20 12:31:17 are_we_quorate(712):1: quorum regained, resuming activity
>notice  Jun 20 12:31:17 quorum_api_set_quorum(148):1: This node is within the primary component and will provide service.
>notice  Jun 20 12:31:17 log_view_list(132):1: Members[4]: 101 102 103 104
>debug   Jun 20 12:31:17 send_library_notification(362):1: sending quorum notification to (nil), length = 64
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[104]: votes: 1, expected: 4 flags: 1
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(494):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 104
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 totemsrp_mcast(2304):14: mcasted message added to pending queue
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1a to 1d
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1b to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1c to pending delivery queue
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_cpg_procjoin(1256):5: got procjoin message from cluster node 101
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1d to pending delivery queue
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 8
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: Yes Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[101]: votes: 1, expected: 4 flags: 0
>debug   Jun 20 12:31:17 decode_flags(554):1: flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceState: No
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(482):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1510):1: ENTERING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1512):1: got nodeinfo message from cluster node 101
>debug   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1517):1: nodeinfo message[0]: votes: 0, expected: 0 flags: 0
>trace   Jun 20 12:31:17 find_node_by_nodeid(479):1: ENTERING find_node_by_nodeid()
>trace   Jun 20 12:31:17 find_node_by_nodeid(487):1: LEAVING find_node_by_nodeid()
>trace   Jun 20 12:31:17 recalculate_quorum(773):1: ENTERING recalculate_quorum()
>trace   Jun 20 12:31:17 get_total_votes(745):1: ENTERING get_total_votes()
>trace   Jun 20 12:31:17 get_total_votes(762):1: LEAVING get_total_votes()
>debug   Jun 20 12:31:17 recalculate_quorum(784):1: total_votes=3, expected_votes=4
>trace   Jun 20 12:31:17 calculate_quorum(599):1: ENTERING calculate_quorum()
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 101 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 103 state=1, votes=1, expected=4
>debug   Jun 20 12:31:17 calculate_quorum(609):1: node 104 state=1, votes=1, expected=4
>trace   Jun 20 12:31:17 calculate_quorum(666):1: LEAVING calculate_quorum()
>trace   Jun 20 12:31:17 are_we_quorate(675):1: ENTERING are_we_quorate()
>trace   Jun 20 12:31:17 get_lowest_node_id(508):1: ENTERING get_lowest_node_id()
>debug   Jun 20 12:31:17 get_lowest_node_id(519):1: lowest node id: 101 us: 101
>trace   Jun 20 12:31:17 get_lowest_node_id(522):1: LEAVING get_lowest_node_id()
>trace   Jun 20 12:31:17 are_we_quorate(735):1: LEAVING are_we_quorate()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1277):1: ENTERING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 votequorum_exec_send_quorum_notification(1321):1: LEAVING votequorum_exec_send_quorum_notification()
>trace   Jun 20 12:31:17 recalculate_quorum(795):1: LEAVING recalculate_quorum()
>trace   Jun 20 12:31:17 message_handler_req_exec_votequorum_nodeinfo(1615):1: LEAVING message_handler_req_exec_votequorum_nodeinfo()
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1b
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1c
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1d
>debug   Jun 20 12:31:17 message_handler_mcast(3929):14: Received ringid(192.168.122.101:67872) seq 1e
>debug   Jun 20 12:31:17 messages_deliver_to_app(3768):14: Delivering 1d to 1e
>debug   Jun 20 12:31:17 messages_deliver_to_app(3837):14: Delivering MCAST message with seq 1e to pending delivery queue
>warning Jun 20 12:31:17 message_handler_req_exec_cpg_mcast(1370):5: Unknown node 66 -> we will not deliver message
>[debug] Free'ing ringbuffer: /dev/shm/qb-create_from_file-header
>[root@pcmk-1 ~]#

Actions: View

Attachments on bug 820821: 583724 | 584502 | 584503 | 584918 | 584944 | 589886 | 590918 | 591194 | 591509 | 591532 | 592063 | 593132 | 593133 | 640873 | 640874 | 640875