Attachment 173681 Details for Bug 251966 – taft-01

taft-01

groupd_taft-01.txt (text/plain), 1.00 MB, created by Corey Marthaler on 2007-08-27 15:09:51 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2007-08-27 15:09:51 UTC

Size: 1.00 MB

patch

obsolete

>1187992210 cman: our nodeid 5 name taft-01.lab.msp.redhat.com quorum 1
>1187992210 setup_cpg groupd_handle 6b8b456700000000
>1187992210 groupd confchg total 3 left 0 joined 1
>1187992210 recv global_id 10005 from 4 cur counter 0
>1187992210 client connection 3
>1187992210 got client 3 setup
>1187992210 setup fence 0
>1187992210 client connection 4
>1187992210 got client 4 setup
>1187992210 setup dlm 1
>1187992210 client connection 5
>1187992210 got client 5 setup
>1187992210 setup gfs 2
>1187992210 groupd confchg total 4 left 0 joined 1
>1187992211 got client 3 join
>1187992211 0:default got join
>1187992211 0:default is cpg client 6 name 0_default handle 327b23c600000001
>1187992211 0:default cpg_join ok
>1187992211 0:default waiting for first cpg event
>1187992211 client connection 7
>1187992211 0:default waiting for first cpg event
>1187992211 got client 7 get_group
>1187992211 0:default waiting for first cpg event
>1187992211 0:default waiting for first cpg event
>1187992211 0:default confchg left 0 joined 1 total 3
>1187992211 0:default process_node_join 5
>1187992211 0:default cpg add node 5 total 1
>1187992211 0:default cpg add node 4 total 2
>1187992211 0:default cpg add node 3 total 3
>1187992211 0:default make_event_id 500030001 nodeid 5 memb_count 3 type 1
>1187992211 0:default queue join event for nodeid 5
>1187992211 0:default process_current_event 500030001 5 JOIN_BEGIN
>1187992211 0:default app node init: add 3 total 1
>1187992211 0:default app node init: add 4 total 2
>1187992211 0:default app node init: add 5 total 3
>1187992211 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 5
>1187992211 0:default mark node 3 stopped
>1187992211 0:default set global_id 10002 from 3
>1187992211 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992211 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992211 0:default mark node 4 stopped
>1187992211 0:default process_current_event 500030001 5 JOIN_ALL_STOPPED
>1187992211 0:default action for app: setid default 65538
>1187992211 0:default action for app: start default 1 2 3 5 4 3
>1187992211 client connection 7
>1187992211 got client 7 get_group
>1187992211 0:default mark node 4 started
>1187992211 client connection 7
>1187992211 got client 7 get_group
>1187992211 client connection 7
>1187992211 got client 7 get_group
>1187992211 0:default mark node 3 started
>1187992211 client connection 7
>1187992211 got client 7 get_group
>1187992211 got client 3 start_done
>1187992211 0:default send started
>1187992211 0:default mark node 5 started
>1187992211 0:default process_current_event 500030001 5 JOIN_ALL_STARTED
>1187992211 0:default action for app: finish default 1
>1187992211 0:default confchg left 0 joined 1 total 4
>1187992211 0:default process_node_join 2
>1187992211 0:default cpg add node 2 total 4
>1187992211 0:default make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992211 0:default queue join event for nodeid 2
>1187992211 0:default process_current_event 200040001 2 JOIN_BEGIN
>1187992211 0:default action for app: stop default
>1187992211 0:default app node join: add 2 total 4
>1187992211 got client 3 stop_done
>1187992211 0:default send stopped
>1187992211 0:default waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992211 0:default mark node 5 stopped
>1187992211 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992211 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992211 0:default mark node 3 stopped
>1187992211 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992211 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992211 0:default mark node 4 stopped
>1187992211 0:default process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992211 0:default action for app: start default 2 2 4 2 5 4 3
>1187992211 got client 3 start_done
>1187992211 0:default send started
>1187992211 0:default mark node 4 started
>1187992211 0:default mark node 5 started
>1187992211 0:default mark node 3 started
>1187992211 0:default mark node 2 started
>1187992211 0:default process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992211 0:default action for app: finish default 2
>1187992212 client connection 7
>1187992212 got client 7 get_group
>1187992213 got client 4 join
>1187992213 1:clvmd got join
>1187992213 1:clvmd is cpg client 7 name 1_clvmd handle 643c986900000002
>1187992213 1:clvmd cpg_join ok
>1187992213 1:clvmd waiting for first cpg event
>1187992213 1:clvmd confchg left 0 joined 1 total 2
>1187992213 1:clvmd process_node_join 5
>1187992213 1:clvmd cpg add node 5 total 1
>1187992213 1:clvmd cpg add node 4 total 2
>1187992213 1:clvmd make_event_id 500020001 nodeid 5 memb_count 2 type 1
>1187992213 1:clvmd queue join event for nodeid 5
>1187992213 1:clvmd process_current_event 500020001 5 JOIN_BEGIN
>1187992213 1:clvmd app node init: add 4 total 1
>1187992213 1:clvmd app node init: add 5 total 2
>1187992213 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992213 1:clvmd mark node 4 stopped
>1187992213 1:clvmd set global_id 10005 from 4
>1187992213 1:clvmd process_current_event 500020001 5 JOIN_ALL_STOPPED
>1187992213 1:clvmd action for app: setid clvmd 65541
>1187992213 1:clvmd action for app: start clvmd 3 2 2 5 4
>1187992213 1:clvmd mark node 4 started
>1187992213 got client 4 start_done
>1187992213 1:clvmd send started
>1187992213 1:clvmd mark node 5 started
>1187992213 1:clvmd process_current_event 500020001 5 JOIN_ALL_STARTED
>1187992213 1:clvmd action for app: finish clvmd 3
>1187992213 1:clvmd confchg left 0 joined 1 total 3
>1187992213 1:clvmd process_node_join 3
>1187992213 1:clvmd cpg add node 3 total 3
>1187992213 1:clvmd make_event_id 300030001 nodeid 3 memb_count 3 type 1
>1187992213 1:clvmd queue join event for nodeid 3
>1187992213 1:clvmd process_current_event 300030001 3 JOIN_BEGIN
>1187992213 1:clvmd action for app: stop clvmd
>1187992213 1:clvmd app node join: add 3 total 3
>1187992213 got client 4 stop_done
>1187992213 1:clvmd send stopped
>1187992213 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992213 1:clvmd mark node 5 stopped
>1187992213 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992213 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992213 1:clvmd mark node 4 stopped
>1187992213 1:clvmd process_current_event 300030001 3 JOIN_ALL_STOPPED
>1187992213 1:clvmd action for app: start clvmd 4 2 3 3 5 4
>1187992213 got client 4 start_done
>1187992213 1:clvmd send started
>1187992213 1:clvmd mark node 4 started
>1187992213 1:clvmd mark node 5 started
>1187992213 1:clvmd mark node 3 started
>1187992213 1:clvmd process_current_event 300030001 3 JOIN_ALL_STARTED
>1187992213 1:clvmd action for app: finish clvmd 4
>1187992213 1:clvmd confchg left 0 joined 1 total 4
>1187992213 1:clvmd process_node_join 2
>1187992213 1:clvmd cpg add node 2 total 4
>1187992213 1:clvmd make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992213 1:clvmd queue join event for nodeid 2
>1187992213 1:clvmd process_current_event 200040001 2 JOIN_BEGIN
>1187992213 1:clvmd action for app: stop clvmd
>1187992213 1:clvmd app node join: add 2 total 4
>1187992213 got client 4 stop_done
>1187992213 1:clvmd send stopped
>1187992213 1:clvmd waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992213 1:clvmd mark node 5 stopped
>1187992213 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992213 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992213 1:clvmd mark node 3 stopped
>1187992213 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992213 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992213 1:clvmd mark node 4 stopped
>1187992213 1:clvmd process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992213 1:clvmd action for app: start clvmd 5 2 4 2 3 5 4
>1187992213 got client 4 start_done
>1187992213 1:clvmd send started
>1187992213 1:clvmd mark node 4 started
>1187992213 1:clvmd mark node 5 started
>1187992213 1:clvmd mark node 2 started
>1187992213 1:clvmd mark node 3 started
>1187992213 1:clvmd process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992213 1:clvmd action for app: finish clvmd 5
>1187992240 client connection 8
>1187992240 got client 8 get_group
>1187992240 got client 5 join
>1187992240 2:gfs1 got join
>1187992240 2:gfs1 is cpg client 8 name 2_gfs1 handle 6633487300000003
>1187992240 2:gfs1 cpg_join ok
>1187992240 2:gfs1 waiting for first cpg event
>1187992240 2:gfs1 confchg left 0 joined 1 total 2
>1187992240 2:gfs1 process_node_join 5
>1187992240 2:gfs1 cpg add node 5 total 1
>1187992240 2:gfs1 cpg add node 4 total 2
>1187992240 2:gfs1 make_event_id 500020001 nodeid 5 memb_count 2 type 1
>1187992240 2:gfs1 queue join event for nodeid 5
>1187992240 2:gfs1 process_current_event 500020001 5 JOIN_BEGIN
>1187992240 2:gfs1 app node init: add 4 total 1
>1187992240 2:gfs1 app node init: add 5 total 2
>1187992240 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992240 2:gfs1 mark node 4 stopped
>1187992240 2:gfs1 set global_id 20002 from 4
>1187992240 2:gfs1 process_current_event 500020001 5 JOIN_ALL_STOPPED
>1187992240 2:gfs1 action for app: setid gfs1 131074
>1187992240 2:gfs1 action for app: start gfs1 6 2 2 5 4
>1187992240 got client 5 start_done
>1187992240 2:gfs1 send started
>1187992240 2:gfs1 mark node 4 started
>1187992240 2:gfs1 mark node 5 started
>1187992240 2:gfs1 process_current_event 500020001 5 JOIN_ALL_STARTED
>1187992240 2:gfs1 action for app: finish gfs1 6
>1187992240 got client 4 join
>1187992240 1:gfs1 got join
>1187992240 1:gfs1 is cpg client 9 name 1_gfs1 handle 74b0dc5100000004
>1187992240 1:gfs1 cpg_join ok
>1187992240 1:gfs1 waiting for first cpg event
>1187992240 1:gfs1 confchg left 0 joined 1 total 2
>1187992240 1:gfs1 process_node_join 5
>1187992240 1:gfs1 cpg add node 5 total 1
>1187992240 1:gfs1 cpg add node 4 total 2
>1187992240 1:gfs1 make_event_id 500020001 nodeid 5 memb_count 2 type 1
>1187992240 1:gfs1 queue join event for nodeid 5
>1187992240 1:gfs1 process_current_event 500020001 5 JOIN_BEGIN
>1187992240 1:gfs1 app node init: add 4 total 1
>1187992240 1:gfs1 app node init: add 5 total 2
>1187992240 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992240 1:gfs1 mark node 4 stopped
>1187992240 1:gfs1 set global_id 30002 from 4
>1187992240 1:gfs1 process_current_event 500020001 5 JOIN_ALL_STOPPED
>1187992240 1:gfs1 action for app: setid gfs1 196610
>1187992240 1:gfs1 action for app: start gfs1 7 2 2 5 4
>1187992240 got client 4 start_done
>1187992240 1:gfs1 send started
>1187992240 1:gfs1 mark node 4 started
>1187992240 1:gfs1 mark node 5 started
>1187992240 1:gfs1 process_current_event 500020001 5 JOIN_ALL_STARTED
>1187992240 1:gfs1 action for app: finish gfs1 7
>1187992240 client connection 10
>1187992240 got client 10 get_group
>1187992240 got client 5 join
>1187992240 2:gfs2 got join
>1187992240 2:gfs2 is cpg client 10 name 2_gfs2 handle 19495cff00000005
>1187992240 2:gfs2 cpg_join ok
>1187992240 2:gfs2 waiting for first cpg event
>1187992240 2:gfs2 confchg left 0 joined 1 total 2
>1187992240 2:gfs2 process_node_join 5
>1187992240 2:gfs2 cpg add node 5 total 1
>1187992240 2:gfs2 cpg add node 4 total 2
>1187992240 2:gfs2 make_event_id 500020001 nodeid 5 memb_count 2 type 1
>1187992240 2:gfs2 queue join event for nodeid 5
>1187992240 2:gfs2 process_current_event 500020001 5 JOIN_BEGIN
>1187992240 2:gfs2 app node init: add 4 total 1
>1187992240 2:gfs2 app node init: add 5 total 2
>1187992240 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992240 2:gfs2 mark node 4 stopped
>1187992240 2:gfs2 set global_id 40002 from 4
>1187992240 2:gfs2 process_current_event 500020001 5 JOIN_ALL_STOPPED
>1187992240 2:gfs2 action for app: setid gfs2 262146
>1187992240 2:gfs2 action for app: start gfs2 8 2 2 5 4
>1187992240 got client 5 start_done
>1187992240 2:gfs2 send started
>1187992240 2:gfs2 mark node 4 started
>1187992240 2:gfs2 mark node 5 started
>1187992240 2:gfs2 process_current_event 500020001 5 JOIN_ALL_STARTED
>1187992240 2:gfs2 action for app: finish gfs2 8
>1187992240 got client 4 join
>1187992240 1:gfs2 got join
>1187992240 1:gfs2 is cpg client 11 name 1_gfs2 handle 2ae8944a00000006
>1187992240 1:gfs2 cpg_join ok
>1187992240 1:gfs2 waiting for first cpg event
>1187992240 1:gfs2 confchg left 0 joined 1 total 2
>1187992240 1:gfs2 process_node_join 5
>1187992240 1:gfs2 cpg add node 5 total 1
>1187992240 1:gfs2 cpg add node 4 total 2
>1187992240 1:gfs2 make_event_id 500020001 nodeid 5 memb_count 2 type 1
>1187992240 1:gfs2 queue join event for nodeid 5
>1187992240 1:gfs2 process_current_event 500020001 5 JOIN_BEGIN
>1187992240 1:gfs2 app node init: add 4 total 1
>1187992240 1:gfs2 app node init: add 5 total 2
>1187992240 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 5
>1187992240 1:gfs2 mark node 4 stopped
>1187992240 1:gfs2 set global_id 50002 from 4
>1187992240 1:gfs2 process_current_event 500020001 5 JOIN_ALL_STOPPED
>1187992240 1:gfs2 action for app: setid gfs2 327682
>1187992240 1:gfs2 action for app: start gfs2 9 2 2 5 4
>1187992240 1:gfs2 mark node 4 started
>1187992240 got client 4 start_done
>1187992240 1:gfs2 send started
>1187992240 1:gfs2 mark node 5 started
>1187992240 1:gfs2 process_current_event 500020001 5 JOIN_ALL_STARTED
>1187992240 1:gfs2 action for app: finish gfs2 9
>1187992240 2:gfs1 confchg left 0 joined 1 total 3
>1187992240 2:gfs1 process_node_join 2
>1187992240 2:gfs1 cpg add node 2 total 3
>1187992240 2:gfs1 make_event_id 200030001 nodeid 2 memb_count 3 type 1
>1187992240 2:gfs1 queue join event for nodeid 2
>1187992240 2:gfs1 process_current_event 200030001 2 JOIN_BEGIN
>1187992240 2:gfs1 action for app: stop gfs1
>1187992240 2:gfs1 app node join: add 2 total 3
>1187992240 got client 5 stop_done
>1187992240 2:gfs1 send stopped
>1187992240 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 2:gfs1 mark node 4 stopped
>1187992240 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 2:gfs1 mark node 5 stopped
>1187992240 2:gfs1 process_current_event 200030001 2 JOIN_ALL_STOPPED
>1187992240 2:gfs1 action for app: start gfs1 10 2 3 2 5 4
>1187992240 got client 5 start_done
>1187992240 2:gfs1 send started
>1187992240 2:gfs1 mark node 4 started
>1187992240 2:gfs1 mark node 5 started
>1187992240 2:gfs1 mark node 2 started
>1187992240 2:gfs1 process_current_event 200030001 2 JOIN_ALL_STARTED
>1187992240 2:gfs1 action for app: finish gfs1 10
>1187992240 1:gfs1 confchg left 0 joined 1 total 3
>1187992240 1:gfs1 process_node_join 2
>1187992240 1:gfs1 cpg add node 2 total 3
>1187992240 1:gfs1 make_event_id 200030001 nodeid 2 memb_count 3 type 1
>1187992240 1:gfs1 queue join event for nodeid 2
>1187992240 1:gfs1 process_current_event 200030001 2 JOIN_BEGIN
>1187992240 1:gfs1 action for app: stop gfs1
>1187992240 1:gfs1 app node join: add 2 total 3
>1187992240 got client 4 stop_done
>1187992240 1:gfs1 send stopped
>1187992240 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 1:gfs1 mark node 5 stopped
>1187992240 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992240 1:gfs1 mark node 4 stopped
>1187992240 1:gfs1 process_current_event 200030001 2 JOIN_ALL_STOPPED
>1187992240 1:gfs1 action for app: start gfs1 11 2 3 2 5 4
>1187992240 got client 4 start_done
>1187992240 1:gfs1 send started
>1187992240 1:gfs1 mark node 4 started
>1187992240 1:gfs1 mark node 5 started
>1187992240 1:gfs1 mark node 2 started
>1187992240 1:gfs1 process_current_event 200030001 2 JOIN_ALL_STARTED
>1187992240 1:gfs1 action for app: finish gfs1 11
>1187992241 2:gfs2 confchg left 0 joined 1 total 3
>1187992241 2:gfs2 process_node_join 2
>1187992241 2:gfs2 cpg add node 2 total 3
>1187992241 2:gfs2 make_event_id 200030001 nodeid 2 memb_count 3 type 1
>1187992241 2:gfs2 queue join event for nodeid 2
>1187992241 2:gfs2 process_current_event 200030001 2 JOIN_BEGIN
>1187992241 2:gfs2 action for app: stop gfs2
>1187992241 2:gfs2 app node join: add 2 total 3
>1187992241 got client 5 stop_done
>1187992241 2:gfs2 send stopped
>1187992241 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 2:gfs2 mark node 5 stopped
>1187992241 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 2:gfs2 mark node 4 stopped
>1187992241 2:gfs2 process_current_event 200030001 2 JOIN_ALL_STOPPED
>1187992241 2:gfs2 action for app: start gfs2 12 2 3 2 5 4
>1187992241 got client 5 start_done
>1187992241 2:gfs2 send started
>1187992241 2:gfs2 mark node 4 started
>1187992241 2:gfs2 mark node 5 started
>1187992241 2:gfs2 mark node 2 started
>1187992241 2:gfs2 process_current_event 200030001 2 JOIN_ALL_STARTED
>1187992241 2:gfs2 action for app: finish gfs2 12
>1187992241 1:gfs2 confchg left 0 joined 1 total 3
>1187992241 1:gfs2 process_node_join 2
>1187992241 1:gfs2 cpg add node 2 total 3
>1187992241 1:gfs2 make_event_id 200030001 nodeid 2 memb_count 3 type 1
>1187992241 1:gfs2 queue join event for nodeid 2
>1187992241 1:gfs2 process_current_event 200030001 2 JOIN_BEGIN
>1187992241 1:gfs2 action for app: stop gfs2
>1187992241 1:gfs2 app node join: add 2 total 3
>1187992241 got client 4 stop_done
>1187992241 1:gfs2 send stopped
>1187992241 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 1:gfs2 mark node 5 stopped
>1187992241 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992241 1:gfs2 mark node 4 stopped
>1187992241 1:gfs2 process_current_event 200030001 2 JOIN_ALL_STOPPED
>1187992241 1:gfs2 action for app: start gfs2 13 2 3 2 5 4
>1187992241 got client 4 start_done
>1187992241 1:gfs2 send started
>1187992241 1:gfs2 mark node 4 started
>1187992241 1:gfs2 mark node 5 started
>1187992241 1:gfs2 mark node 2 started
>1187992241 1:gfs2 process_current_event 200030001 2 JOIN_ALL_STARTED
>1187992241 1:gfs2 action for app: finish gfs2 13
>1187992242 2:gfs1 confchg left 0 joined 1 total 4
>1187992242 2:gfs1 process_node_join 3
>1187992242 2:gfs1 cpg add node 3 total 4
>1187992242 2:gfs1 make_event_id 300040001 nodeid 3 memb_count 4 type 1
>1187992242 2:gfs1 queue join event for nodeid 3
>1187992242 2:gfs1 process_current_event 300040001 3 JOIN_BEGIN
>1187992242 2:gfs1 action for app: stop gfs1
>1187992242 2:gfs1 app node join: add 3 total 4
>1187992242 got client 5 stop_done
>1187992242 2:gfs1 send stopped
>1187992242 2:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs1 mark node 2 stopped
>1187992242 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs1 mark node 4 stopped
>1187992242 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs1 mark node 5 stopped
>1187992242 2:gfs1 process_current_event 300040001 3 JOIN_ALL_STOPPED
>1187992242 2:gfs1 action for app: start gfs1 14 2 4 3 2 5 4
>1187992242 got client 5 start_done
>1187992242 2:gfs1 send started
>1187992242 2:gfs1 mark node 4 started
>1187992242 2:gfs1 mark node 5 started
>1187992242 2:gfs1 mark node 2 started
>1187992242 2:gfs1 mark node 3 started
>1187992242 2:gfs1 process_current_event 300040001 3 JOIN_ALL_STARTED
>1187992242 2:gfs1 action for app: finish gfs1 14
>1187992242 1:gfs1 confchg left 0 joined 1 total 4
>1187992242 1:gfs1 process_node_join 3
>1187992242 1:gfs1 cpg add node 3 total 4
>1187992242 1:gfs1 make_event_id 300040001 nodeid 3 memb_count 4 type 1
>1187992242 1:gfs1 queue join event for nodeid 3
>1187992242 1:gfs1 process_current_event 300040001 3 JOIN_BEGIN
>1187992242 1:gfs1 action for app: stop gfs1
>1187992242 1:gfs1 app node join: add 3 total 4
>1187992242 got client 4 stop_done
>1187992242 1:gfs1 send stopped
>1187992242 1:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs1 mark node 2 stopped
>1187992242 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs1 mark node 4 stopped
>1187992242 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs1 mark node 5 stopped
>1187992242 1:gfs1 process_current_event 300040001 3 JOIN_ALL_STOPPED
>1187992242 1:gfs1 action for app: start gfs1 15 2 4 3 2 5 4
>1187992242 got client 4 start_done
>1187992242 1:gfs1 send started
>1187992242 1:gfs1 mark node 5 started
>1187992242 1:gfs1 mark node 2 started
>1187992242 1:gfs1 mark node 4 started
>1187992242 1:gfs1 mark node 3 started
>1187992242 1:gfs1 process_current_event 300040001 3 JOIN_ALL_STARTED
>1187992242 1:gfs1 action for app: finish gfs1 15
>1187992242 2:gfs2 confchg left 0 joined 1 total 4
>1187992242 2:gfs2 process_node_join 3
>1187992242 2:gfs2 cpg add node 3 total 4
>1187992242 2:gfs2 make_event_id 300040001 nodeid 3 memb_count 4 type 1
>1187992242 2:gfs2 queue join event for nodeid 3
>1187992242 2:gfs2 process_current_event 300040001 3 JOIN_BEGIN
>1187992242 2:gfs2 action for app: stop gfs2
>1187992242 2:gfs2 app node join: add 3 total 4
>1187992242 got client 5 stop_done
>1187992242 2:gfs2 send stopped
>1187992242 2:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs2 mark node 2 stopped
>1187992242 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs2 mark node 4 stopped
>1187992242 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 2:gfs2 mark node 5 stopped
>1187992242 2:gfs2 process_current_event 300040001 3 JOIN_ALL_STOPPED
>1187992242 2:gfs2 action for app: start gfs2 16 2 4 3 2 5 4
>1187992242 got client 5 start_done
>1187992242 2:gfs2 send started
>1187992242 2:gfs2 mark node 4 started
>1187992242 2:gfs2 mark node 5 started
>1187992242 2:gfs2 mark node 2 started
>1187992242 2:gfs2 mark node 3 started
>1187992242 2:gfs2 process_current_event 300040001 3 JOIN_ALL_STARTED
>1187992242 2:gfs2 action for app: finish gfs2 16
>1187992242 1:gfs2 confchg left 0 joined 1 total 4
>1187992242 1:gfs2 process_node_join 3
>1187992242 1:gfs2 cpg add node 3 total 4
>1187992242 1:gfs2 make_event_id 300040001 nodeid 3 memb_count 4 type 1
>1187992242 1:gfs2 queue join event for nodeid 3
>1187992242 1:gfs2 process_current_event 300040001 3 JOIN_BEGIN
>1187992242 1:gfs2 action for app: stop gfs2
>1187992242 1:gfs2 app node join: add 3 total 4
>1187992242 got client 4 stop_done
>1187992242 1:gfs2 send stopped
>1187992242 1:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs2 mark node 5 stopped
>1187992242 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs2 mark node 2 stopped
>1187992242 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187992242 1:gfs2 mark node 4 stopped
>1187992242 1:gfs2 process_current_event 300040001 3 JOIN_ALL_STOPPED
>1187992242 1:gfs2 action for app: start gfs2 17 2 4 3 2 5 4
>1187992242 got client 4 start_done
>1187992242 1:gfs2 send started
>1187992242 1:gfs2 mark node 4 started
>1187992242 1:gfs2 mark node 5 started
>1187992242 1:gfs2 mark node 2 started
>1187992242 1:gfs2 mark node 3 started
>1187992242 1:gfs2 process_current_event 300040001 3 JOIN_ALL_STARTED
>1187992242 1:gfs2 action for app: finish gfs2 17
>1187992352 cman: node 4 removed
>1187992352 add_recovery_set_cman nodeid 4
>1187992352 groupd confchg total 3 left 1 joined 0
>1187992352 add_recovery_set_cpg nodeid 4 procdown 0
>1187992352 0:default add to recovery set 4
>1187992352 1:clvmd add to recovery set 4
>1187992352 2:gfs1 add to recovery set 4
>1187992352 1:gfs1 add to recovery set 4
>1187992352 2:gfs2 add to recovery set 4
>1187992352 1:gfs2 add to recovery set 4
>1187992352 0:default confchg left 1 joined 0 total 3
>1187992352 0:default confchg removed node 4 reason 3
>1187992352 0:default process_node_down 4
>1187992352 0:default cpg del node 4 total 3 - down
>1187992352 0:default make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 0:default queue recover event for nodeid 4
>1187992352 2:gfs1 confchg left 1 joined 0 total 3
>1187992352 2:gfs1 confchg removed node 4 reason 3
>1187992352 2:gfs1 process_node_down 4
>1187992352 2:gfs1 cpg del node 4 total 3 - down
>1187992352 2:gfs1 make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 2:gfs1 queue recover event for nodeid 4
>1187992352 1:gfs1 confchg left 1 joined 0 total 3
>1187992352 1:gfs1 confchg removed node 4 reason 3
>1187992352 1:gfs1 process_node_down 4
>1187992352 1:gfs1 cpg del node 4 total 3 - down
>1187992352 1:gfs1 make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 1:gfs1 queue recover event for nodeid 4
>1187992352 1:gfs2 confchg left 1 joined 0 total 3
>1187992352 1:gfs2 confchg removed node 4 reason 3
>1187992352 1:gfs2 process_node_down 4
>1187992352 1:gfs2 cpg del node 4 total 3 - down
>1187992352 1:gfs2 make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 1:gfs2 queue recover event for nodeid 4
>1187992352 0:default set current event to recovery for 4
>1187992352 0:default process_current_event 400030003 4 FAIL_BEGIN
>1187992352 0:default action for app: stop default
>1187992352 2:gfs1 set current event to recovery for 4
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_BEGIN
>1187992352 2:gfs1 action for app: stop gfs1
>1187992352 1:gfs1 set current event to recovery for 4
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_BEGIN
>1187992352 1:gfs1 action for app: stop gfs1
>1187992352 1:gfs2 set current event to recovery for 4
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_BEGIN
>1187992352 1:gfs2 action for app: stop gfs2
>1187992352 1:clvmd confchg left 1 joined 0 total 3
>1187992352 1:clvmd confchg removed node 4 reason 3
>1187992352 1:clvmd process_node_down 4
>1187992352 1:clvmd cpg del node 4 total 3 - down
>1187992352 1:clvmd make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 1:clvmd queue recover event for nodeid 4
>1187992352 2:gfs2 confchg left 1 joined 0 total 3
>1187992352 2:gfs2 confchg removed node 4 reason 3
>1187992352 2:gfs2 process_node_down 4
>1187992352 2:gfs2 cpg del node 4 total 3 - down
>1187992352 2:gfs2 make_event_id 400030003 nodeid 4 memb_count 3 type 3
>1187992352 2:gfs2 queue recover event for nodeid 4
>1187992352 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd set current event to recovery for 4
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_BEGIN
>1187992352 1:clvmd action for app: stop clvmd
>1187992352 2:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 set current event to recovery for 4
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_BEGIN
>1187992352 2:gfs2 action for app: stop gfs2
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 got client 5 stop_done
>1187992352 2:gfs1 send stopped
>1187992352 2:gfs1 cpg_mcast_joined retried 5
>1187992352 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 got client 3 stop_done
>1187992352 0:default send stopped
>1187992352 got client 4 stop_done
>1187992352 1:gfs1 send stopped
>1187992352 got client 5 stop_done
>1187992352 2:gfs2 send stopped
>1187992352 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 got client 4 stop_done
>1187992352 1:gfs2 send stopped
>1187992352 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 mark node 5 stopped
>1187992352 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 got client 4 stop_done
>1187992352 1:clvmd send stopped
>1187992352 0:default mark node 5 stopped
>1187992352 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 mark node 3 stopped
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default mark node 3 stopped
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd mark node 3 stopped
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 mark node 3 stopped
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 mark node 3 stopped
>1187992352 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 mark node 5 stopped
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 mark node 5 stopped
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 mark node 5 stopped
>1187992352 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd mark node 5 stopped
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 mark node 3 stopped
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default mark node 2 stopped
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd mark node 2 stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 mark node 2 stopped
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 mark node 2 stopped
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 mark node 2 stopped
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 4
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default wait for all_levels_all_stopped
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 mark node 2 stopped
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 0:default process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 0:default app node fail: del node 4 total 3
>1187992352 0:default action for app: start default 18 1 3 2 5 3
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 got client 3 start_done
>1187992352 0:default send started
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 0:default mark node 3 started
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 0:default mark node 5 started
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992352 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:clvmd lower levels not recovered in rs 4
>1187992352 1:clvmd wait for lower_levels_recovered
>1187992352 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs1 lower levels not recovered in rs 4
>1187992352 2:gfs1 wait for lower_levels_recovered
>1187992352 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs1 lower levels not recovered in rs 4
>1187992352 1:gfs1 wait for lower_levels_recovered
>1187992352 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 2:gfs2 lower levels not recovered in rs 4
>1187992352 2:gfs2 wait for lower_levels_recovered
>1187992352 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992352 1:gfs2 lower levels not recovered in rs 4
>1187992352 1:gfs2 wait for lower_levels_recovered
>1187992387 0:default mark node 2 started
>1187992387 0:default process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992387 0:default action for app: finish default 18
>1187992387 0:default rev 400030003 done, remove group from rs 4
>1187992387 0:default done in recovery set 4
>1187992387 recovery set 4 has 5 entries not done
>1187992387 1:clvmd process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 1:clvmd app node fail: del node 4 total 3
>1187992387 1:clvmd action for app: start clvmd 22 1 3 2 3 5
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 1:gfs1 app node fail: del node 4 total 3
>1187992387 1:gfs1 action for app: start gfs1 20 1 3 3 2 5
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 1:gfs2 app node fail: del node 4 total 3
>1187992387 1:gfs2 action for app: start gfs2 21 1 3 3 2 5
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 got client 4 start_done
>1187992387 1:clvmd send started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 got client 4 start_done
>1187992387 1:gfs1 send started
>1187992387 1:clvmd mark node 3 started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 got client 4 start_done
>1187992387 1:gfs2 send started
>1187992387 1:clvmd mark node 5 started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 1:clvmd mark node 2 started
>1187992387 1:clvmd process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992387 1:clvmd action for app: finish clvmd 22
>1187992387 1:clvmd rev 400030003 done, remove group from rs 4
>1187992387 1:clvmd done in recovery set 4
>1187992387 recovery set 4 has 4 entries not done
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 1:gfs1 mark node 2 started
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 1:gfs2 mark node 2 started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 1:gfs1 mark node 3 started
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 1:gfs2 mark node 3 started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 1:gfs1 mark node 5 started
>1187992387 1:gfs1 process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992387 1:gfs1 action for app: finish gfs1 20
>1187992387 1:gfs1 rev 400030003 done, remove group from rs 4
>1187992387 1:gfs1 done in recovery set 4
>1187992387 recovery set 4 has 3 entries not done
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 lower levels not recovered in rs 4
>1187992387 2:gfs1 wait for lower_levels_recovered
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 lower levels not recovered in rs 4
>1187992387 2:gfs2 wait for lower_levels_recovered
>1187992387 1:gfs2 mark node 5 started
>1187992387 1:gfs2 process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992387 1:gfs2 action for app: finish gfs2 21
>1187992387 1:gfs2 rev 400030003 done, remove group from rs 4
>1187992387 1:gfs2 done in recovery set 4
>1187992387 recovery set 4 has 2 entries not done
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs1 app node fail: del node 4 total 3
>1187992387 2:gfs1 action for app: start gfs1 19 1 3 3 2 5
>1187992387 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STOPPED
>1187992387 2:gfs2 app node fail: del node 4 total 3
>1187992387 2:gfs2 action for app: start gfs2 23 1 3 3 2 5
>1187992387 got client 5 start_done
>1187992387 2:gfs1 send started
>1187992387 2:gfs1 mark node 5 started
>1187992387 2:gfs1 mark node 3 started
>1187992387 2:gfs1 mark node 2 started
>1187992387 2:gfs1 process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992387 2:gfs1 action for app: finish gfs1 19
>1187992387 2:gfs1 rev 400030003 done, remove group from rs 4
>1187992387 2:gfs1 done in recovery set 4
>1187992387 recovery set 4 has 1 entries not done
>1187992388 2:gfs2 mark node 3 started
>1187992388 got client 5 start_done
>1187992388 2:gfs2 send started
>1187992388 2:gfs2 mark node 5 started
>1187992388 2:gfs2 mark node 2 started
>1187992388 2:gfs2 process_current_event 400030003 4 FAIL_ALL_STARTED
>1187992388 2:gfs2 action for app: finish gfs2 23
>1187992388 2:gfs2 rev 400030003 done, remove group from rs 4
>1187992388 2:gfs2 done in recovery set 4
>1187992388 recovery set 4 is all done
>1187992498 cman: node 4 added
>1187992499 groupd confchg total 4 left 0 joined 1
>1187992500 0:default confchg left 0 joined 1 total 4
>1187992500 0:default process_node_join 4
>1187992500 0:default cpg add node 4 total 4
>1187992500 0:default make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992500 0:default queue join event for nodeid 4
>1187992500 0:default process_current_event 400040001 4 JOIN_BEGIN
>1187992500 0:default action for app: stop default
>1187992500 0:default app node join: add 4 total 4
>1187992500 got client 3 stop_done
>1187992500 0:default send stopped
>1187992500 0:default waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992500 0:default mark node 3 stopped
>1187992500 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992500 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992500 0:default mark node 5 stopped
>1187992500 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992500 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992500 0:default mark node 2 stopped
>1187992500 0:default process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992500 0:default action for app: start default 24 2 4 4 2 5 3
>1187992500 got client 3 start_done
>1187992500 0:default send started
>1187992500 0:default mark node 2 started
>1187992500 0:default mark node 3 started
>1187992500 0:default mark node 5 started
>1187992500 0:default mark node 4 started
>1187992500 0:default process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992500 0:default action for app: finish default 24
>1187992504 1:clvmd confchg left 0 joined 1 total 4
>1187992504 1:clvmd process_node_join 4
>1187992504 1:clvmd cpg add node 4 total 4
>1187992504 1:clvmd make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992504 1:clvmd queue join event for nodeid 4
>1187992504 1:clvmd process_current_event 400040001 4 JOIN_BEGIN
>1187992504 1:clvmd action for app: stop clvmd
>1187992504 1:clvmd app node join: add 4 total 4
>1187992504 got client 4 stop_done
>1187992504 1:clvmd send stopped
>1187992504 1:clvmd waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992504 1:clvmd mark node 3 stopped
>1187992504 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992504 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992504 1:clvmd mark node 5 stopped
>1187992504 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992504 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992504 1:clvmd mark node 2 stopped
>1187992504 1:clvmd process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992504 1:clvmd action for app: start clvmd 25 2 4 4 2 3 5
>1187992504 got client 4 start_done
>1187992504 1:clvmd send started
>1187992504 1:clvmd mark node 5 started
>1187992504 1:clvmd mark node 2 started
>1187992504 1:clvmd mark node 3 started
>1187992504 1:clvmd mark node 4 started
>1187992504 1:clvmd process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992504 1:clvmd action for app: finish clvmd 25
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992515 client connection 12
>1187992515 got client 12 get_groups
>1187992516 client connection 12
>1187992516 got client 12 get_groups
>1187992516 2:gfs1 confchg left 0 joined 1 total 4
>1187992516 2:gfs1 process_node_join 4
>1187992516 2:gfs1 cpg add node 4 total 4
>1187992516 2:gfs1 make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992516 2:gfs1 queue join event for nodeid 4
>1187992516 2:gfs1 process_current_event 400040001 4 JOIN_BEGIN
>1187992516 2:gfs1 action for app: stop gfs1
>1187992516 2:gfs1 app node join: add 4 total 4
>1187992516 got client 5 stop_done
>1187992516 2:gfs1 send stopped
>1187992516 2:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs1 mark node 3 stopped
>1187992516 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs1 mark node 5 stopped
>1187992516 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs1 mark node 2 stopped
>1187992516 2:gfs1 process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992516 2:gfs1 action for app: start gfs1 26 2 4 4 3 2 5
>1187992516 got client 5 start_done
>1187992516 2:gfs1 send started
>1187992516 2:gfs1 mark node 5 started
>1187992516 2:gfs1 mark node 2 started
>1187992516 2:gfs1 mark node 3 started
>1187992516 2:gfs1 mark node 4 started
>1187992516 2:gfs1 process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992516 2:gfs1 action for app: finish gfs1 26
>1187992516 1:gfs1 confchg left 0 joined 1 total 4
>1187992516 1:gfs1 process_node_join 4
>1187992516 1:gfs1 cpg add node 4 total 4
>1187992516 1:gfs1 make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992516 1:gfs1 queue join event for nodeid 4
>1187992516 1:gfs1 process_current_event 400040001 4 JOIN_BEGIN
>1187992516 1:gfs1 action for app: stop gfs1
>1187992516 1:gfs1 app node join: add 4 total 4
>1187992516 got client 4 stop_done
>1187992516 1:gfs1 send stopped
>1187992516 1:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs1 mark node 3 stopped
>1187992516 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs1 mark node 5 stopped
>1187992516 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs1 mark node 2 stopped
>1187992516 1:gfs1 process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992516 1:gfs1 action for app: start gfs1 27 2 4 4 3 2 5
>1187992516 got client 4 start_done
>1187992516 1:gfs1 send started
>1187992516 1:gfs1 mark node 2 started
>1187992516 1:gfs1 mark node 3 started
>1187992516 1:gfs1 mark node 5 started
>1187992516 1:gfs1 mark node 4 started
>1187992516 1:gfs1 process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992516 1:gfs1 action for app: finish gfs1 27
>1187992516 2:gfs2 confchg left 0 joined 1 total 4
>1187992516 2:gfs2 process_node_join 4
>1187992516 2:gfs2 cpg add node 4 total 4
>1187992516 2:gfs2 make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992516 2:gfs2 queue join event for nodeid 4
>1187992516 2:gfs2 process_current_event 400040001 4 JOIN_BEGIN
>1187992516 2:gfs2 action for app: stop gfs2
>1187992516 2:gfs2 app node join: add 4 total 4
>1187992516 got client 5 stop_done
>1187992516 2:gfs2 send stopped
>1187992516 2:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs2 mark node 3 stopped
>1187992516 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs2 mark node 5 stopped
>1187992516 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 2:gfs2 mark node 2 stopped
>1187992516 2:gfs2 process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992516 2:gfs2 action for app: start gfs2 28 2 4 4 3 2 5
>1187992516 got client 5 start_done
>1187992516 2:gfs2 send started
>1187992516 2:gfs2 mark node 5 started
>1187992516 2:gfs2 mark node 2 started
>1187992516 2:gfs2 mark node 3 started
>1187992516 2:gfs2 mark node 4 started
>1187992516 2:gfs2 process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992516 2:gfs2 action for app: finish gfs2 28
>1187992516 1:gfs2 confchg left 0 joined 1 total 4
>1187992516 1:gfs2 process_node_join 4
>1187992516 1:gfs2 cpg add node 4 total 4
>1187992516 1:gfs2 make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187992516 1:gfs2 queue join event for nodeid 4
>1187992516 1:gfs2 process_current_event 400040001 4 JOIN_BEGIN
>1187992516 1:gfs2 action for app: stop gfs2
>1187992516 1:gfs2 app node join: add 4 total 4
>1187992516 got client 4 stop_done
>1187992516 1:gfs2 send stopped
>1187992516 1:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs2 mark node 2 stopped
>1187992516 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs2 mark node 3 stopped
>1187992516 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187992516 1:gfs2 mark node 5 stopped
>1187992516 1:gfs2 process_current_event 400040001 4 JOIN_ALL_STOPPED
>1187992516 1:gfs2 action for app: start gfs2 29 2 4 4 3 2 5
>1187992516 got client 4 start_done
>1187992516 1:gfs2 send started
>1187992516 1:gfs2 mark node 5 started
>1187992516 1:gfs2 mark node 2 started
>1187992516 1:gfs2 mark node 3 started
>1187992516 1:gfs2 mark node 4 started
>1187992516 1:gfs2 process_current_event 400040001 4 JOIN_ALL_STARTED
>1187992516 1:gfs2 action for app: finish gfs2 29
>1187992610 cman: node 2 removed
>1187992610 add_recovery_set_cman nodeid 2
>1187992610 groupd confchg total 3 left 1 joined 0
>1187992610 add_recovery_set_cpg nodeid 2 procdown 0
>1187992610 0:default add to recovery set 2
>1187992610 1:clvmd add to recovery set 2
>1187992610 2:gfs1 add to recovery set 2
>1187992610 1:gfs1 add to recovery set 2
>1187992610 2:gfs2 add to recovery set 2
>1187992610 1:gfs2 add to recovery set 2
>1187992610 0:default confchg left 1 joined 0 total 3
>1187992610 0:default confchg removed node 2 reason 3
>1187992610 0:default process_node_down 2
>1187992610 0:default cpg del node 2 total 3 - down
>1187992610 0:default make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 0:default queue recover event for nodeid 2
>1187992610 1:clvmd confchg left 1 joined 0 total 3
>1187992610 1:clvmd confchg removed node 2 reason 3
>1187992610 1:clvmd process_node_down 2
>1187992610 1:clvmd cpg del node 2 total 3 - down
>1187992610 1:clvmd make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 1:clvmd queue recover event for nodeid 2
>1187992610 2:gfs1 confchg left 1 joined 0 total 3
>1187992610 2:gfs1 confchg removed node 2 reason 3
>1187992610 2:gfs1 process_node_down 2
>1187992610 2:gfs1 cpg del node 2 total 3 - down
>1187992610 2:gfs1 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 2:gfs1 queue recover event for nodeid 2
>1187992610 1:gfs1 confchg left 1 joined 0 total 3
>1187992610 1:gfs1 confchg removed node 2 reason 3
>1187992610 1:gfs1 process_node_down 2
>1187992610 1:gfs1 cpg del node 2 total 3 - down
>1187992610 1:gfs1 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 1:gfs1 queue recover event for nodeid 2
>1187992610 2:gfs2 confchg left 1 joined 0 total 3
>1187992610 2:gfs2 confchg removed node 2 reason 3
>1187992610 2:gfs2 process_node_down 2
>1187992610 2:gfs2 cpg del node 2 total 3 - down
>1187992610 2:gfs2 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 2:gfs2 queue recover event for nodeid 2
>1187992610 1:gfs2 confchg left 1 joined 0 total 3
>1187992610 1:gfs2 confchg removed node 2 reason 3
>1187992610 1:gfs2 process_node_down 2
>1187992610 1:gfs2 cpg del node 2 total 3 - down
>1187992610 1:gfs2 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992610 1:gfs2 queue recover event for nodeid 2
>1187992610 0:default set current event to recovery for 2
>1187992610 0:default process_current_event 200030003 2 FAIL_BEGIN
>1187992610 0:default action for app: stop default
>1187992610 1:clvmd set current event to recovery for 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_BEGIN
>1187992610 1:clvmd action for app: stop clvmd
>1187992610 2:gfs1 set current event to recovery for 2
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_BEGIN
>1187992610 2:gfs1 action for app: stop gfs1
>1187992610 1:gfs1 set current event to recovery for 2
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_BEGIN
>1187992610 1:gfs1 action for app: stop gfs1
>1187992610 2:gfs2 set current event to recovery for 2
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_BEGIN
>1187992610 2:gfs2 action for app: stop gfs2
>1187992610 1:gfs2 set current event to recovery for 2
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_BEGIN
>1187992610 1:gfs2 action for app: stop gfs2
>1187992610 got client 4 stop_done
>1187992610 1:clvmd send stopped
>1187992610 1:clvmd cpg_mcast_joined retried 7
>1187992610 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 got client 3 stop_done
>1187992610 0:default send stopped
>1187992610 got client 4 stop_done
>1187992610 1:gfs1 send stopped
>1187992610 got client 5 stop_done
>1187992610 2:gfs1 send stopped
>1187992610 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 mark node 4 stopped
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 got client 4 stop_done
>1187992610 1:gfs2 send stopped
>1187992610 got client 5 stop_done
>1187992610 2:gfs2 send stopped
>1187992610 0:default mark node 4 stopped
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 mark node 4 stopped
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 mark node 4 stopped
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 mark node 4 stopped
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 3 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd mark node 4 stopped
>1187992610 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd mark node 5 stopped
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 mark node 3 stopped
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default mark node 3 stopped
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd mark node 3 stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 mark node 3 stopped
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 mark node 3 stopped
>1187992610 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default mark node 5 stopped
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 mark node 5 stopped
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 mark node 5 stopped
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 2 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 mark node 5 stopped
>1187992610 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 mark node 5 stopped
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default wait for all_levels_all_stopped
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 mark node 3 stopped
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 0:default app node fail: del node 2 total 3
>1187992610 0:default action for app: start default 30 1 3 4 5 3
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 got client 3 start_done
>1187992610 0:default send started
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 0:default mark node 5 started
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 0:default mark node 4 started
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992610 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:clvmd lower levels not recovered in rs 2
>1187992610 1:clvmd wait for lower_levels_recovered
>1187992610 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs1 lower levels not recovered in rs 2
>1187992610 2:gfs1 wait for lower_levels_recovered
>1187992610 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs1 lower levels not recovered in rs 2
>1187992610 1:gfs1 wait for lower_levels_recovered
>1187992610 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 2:gfs2 lower levels not recovered in rs 2
>1187992610 2:gfs2 wait for lower_levels_recovered
>1187992610 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992610 1:gfs2 lower levels not recovered in rs 2
>1187992610 1:gfs2 wait for lower_levels_recovered
>1187992645 0:default mark node 3 started
>1187992645 0:default process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992645 0:default action for app: finish default 30
>1187992645 0:default rev 200030003 done, remove group from rs 2
>1187992645 0:default done in recovery set 2
>1187992645 recovery set 2 has 5 entries not done
>1187992645 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 1:clvmd app node fail: del node 2 total 3
>1187992645 1:clvmd action for app: start clvmd 31 1 3 4 3 5
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 1:gfs1 app node fail: del node 2 total 3
>1187992645 1:gfs1 action for app: start gfs1 33 1 3 4 3 5
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 1:gfs2 app node fail: del node 2 total 3
>1187992645 1:gfs2 action for app: start gfs2 35 1 3 4 3 5
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 got client 4 start_done
>1187992645 1:clvmd send started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 got client 4 start_done
>1187992645 1:gfs1 send started
>1187992645 1:clvmd mark node 3 started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 got client 4 start_done
>1187992645 1:gfs2 send started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:clvmd mark node 5 started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 1:gfs1 mark node 5 started
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:gfs2 mark node 5 started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 1:gfs1 mark node 3 started
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:gfs2 mark node 3 started
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:clvmd mark node 4 started
>1187992645 1:clvmd process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992645 1:clvmd action for app: finish clvmd 31
>1187992645 1:clvmd rev 200030003 done, remove group from rs 2
>1187992645 1:clvmd done in recovery set 2
>1187992645 recovery set 2 has 4 entries not done
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 1:gfs1 mark node 4 started
>1187992645 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992645 1:gfs1 action for app: finish gfs1 33
>1187992645 1:gfs1 rev 200030003 done, remove group from rs 2
>1187992645 1:gfs1 done in recovery set 2
>1187992645 recovery set 2 has 3 entries not done
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 lower levels not recovered in rs 2
>1187992645 2:gfs1 wait for lower_levels_recovered
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 lower levels not recovered in rs 2
>1187992645 2:gfs2 wait for lower_levels_recovered
>1187992645 1:gfs2 mark node 4 started
>1187992645 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992645 1:gfs2 action for app: finish gfs2 35
>1187992645 1:gfs2 rev 200030003 done, remove group from rs 2
>1187992645 1:gfs2 done in recovery set 2
>1187992645 recovery set 2 has 2 entries not done
>1187992645 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs1 app node fail: del node 2 total 3
>1187992645 2:gfs1 action for app: start gfs1 32 1 3 4 3 5
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992645 2:gfs2 app node fail: del node 2 total 3
>1187992645 2:gfs2 action for app: start gfs2 34 1 3 4 3 5
>1187992645 2:gfs2 mark node 4 started
>1187992645 got client 5 start_done
>1187992645 2:gfs2 send started
>1187992645 2:gfs2 mark node 5 started
>1187992645 got client 5 start_done
>1187992645 2:gfs1 send started
>1187992645 2:gfs1 mark node 5 started
>1187992645 2:gfs1 mark node 4 started
>1187992645 2:gfs2 mark node 3 started
>1187992645 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992645 2:gfs2 action for app: finish gfs2 34
>1187992645 2:gfs2 rev 200030003 done, remove group from rs 2
>1187992645 2:gfs2 done in recovery set 2
>1187992645 recovery set 2 has 1 entries not done
>1187992646 2:gfs1 mark node 3 started
>1187992646 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187992646 2:gfs1 action for app: finish gfs1 32
>1187992646 2:gfs1 rev 200030003 done, remove group from rs 2
>1187992646 2:gfs1 done in recovery set 2
>1187992646 recovery set 2 is all done
>1187992754 cman: node 2 added
>1187992755 groupd confchg total 4 left 0 joined 1
>1187992755 joined node 2 had old max gid 50002
>1187992756 0:default confchg left 0 joined 1 total 4
>1187992756 0:default process_node_join 2
>1187992756 0:default cpg add node 2 total 4
>1187992756 0:default make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992756 0:default queue join event for nodeid 2
>1187992756 0:default process_current_event 200040001 2 JOIN_BEGIN
>1187992756 0:default action for app: stop default
>1187992756 0:default app node join: add 2 total 4
>1187992756 got client 3 stop_done
>1187992756 0:default send stopped
>1187992756 0:default waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992756 0:default mark node 5 stopped
>1187992756 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992756 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992756 0:default mark node 3 stopped
>1187992756 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992756 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992756 0:default mark node 4 stopped
>1187992756 0:default process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992756 0:default action for app: start default 36 2 4 2 4 5 3
>1187992756 got client 3 start_done
>1187992756 0:default send started
>1187992756 0:default mark node 4 started
>1187992756 0:default mark node 5 started
>1187992756 0:default mark node 3 started
>1187992756 0:default mark node 2 started
>1187992756 0:default process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992756 0:default action for app: finish default 36
>1187992760 1:clvmd confchg left 0 joined 1 total 4
>1187992760 1:clvmd process_node_join 2
>1187992760 1:clvmd cpg add node 2 total 4
>1187992760 1:clvmd make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992760 1:clvmd queue join event for nodeid 2
>1187992760 1:clvmd process_current_event 200040001 2 JOIN_BEGIN
>1187992760 1:clvmd action for app: stop clvmd
>1187992760 1:clvmd app node join: add 2 total 4
>1187992760 got client 4 stop_done
>1187992760 1:clvmd send stopped
>1187992760 1:clvmd waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992760 1:clvmd mark node 5 stopped
>1187992760 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992760 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992760 1:clvmd mark node 3 stopped
>1187992760 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992760 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992760 1:clvmd mark node 4 stopped
>1187992760 1:clvmd process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992760 1:clvmd action for app: start clvmd 37 2 4 2 4 3 5
>1187992760 got client 4 start_done
>1187992760 1:clvmd send started
>1187992760 1:clvmd mark node 5 started
>1187992760 1:clvmd mark node 3 started
>1187992760 1:clvmd mark node 4 started
>1187992760 1:clvmd mark node 2 started
>1187992760 1:clvmd process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992760 1:clvmd action for app: finish clvmd 37
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992773 client connection 12
>1187992773 got client 12 get_groups
>1187992774 2:gfs1 confchg left 0 joined 1 total 4
>1187992774 2:gfs1 process_node_join 2
>1187992774 2:gfs1 cpg add node 2 total 4
>1187992774 2:gfs1 make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992774 2:gfs1 queue join event for nodeid 2
>1187992774 2:gfs1 process_current_event 200040001 2 JOIN_BEGIN
>1187992774 2:gfs1 action for app: stop gfs1
>1187992774 2:gfs1 app node join: add 2 total 4
>1187992774 got client 5 stop_done
>1187992774 2:gfs1 send stopped
>1187992774 2:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 2:gfs1 mark node 5 stopped
>1187992774 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 2:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 2:gfs1 mark node 3 stopped
>1187992774 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 2:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 2:gfs1 mark node 4 stopped
>1187992774 2:gfs1 process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992774 2:gfs1 action for app: start gfs1 38 2 4 2 4 3 5
>1187992774 got client 5 start_done
>1187992774 2:gfs1 send started
>1187992774 2:gfs1 mark node 3 started
>1187992774 2:gfs1 mark node 4 started
>1187992774 2:gfs1 mark node 5 started
>1187992774 2:gfs1 mark node 2 started
>1187992774 2:gfs1 process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992774 2:gfs1 action for app: finish gfs1 38
>1187992774 1:gfs1 confchg left 0 joined 1 total 4
>1187992774 1:gfs1 process_node_join 2
>1187992774 1:gfs1 cpg add node 2 total 4
>1187992774 1:gfs1 make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992774 1:gfs1 queue join event for nodeid 2
>1187992774 1:gfs1 process_current_event 200040001 2 JOIN_BEGIN
>1187992774 1:gfs1 action for app: stop gfs1
>1187992774 1:gfs1 app node join: add 2 total 4
>1187992774 got client 4 stop_done
>1187992774 1:gfs1 send stopped
>1187992774 1:gfs1 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 1:gfs1 mark node 5 stopped
>1187992774 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 1:gfs1 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 1:gfs1 mark node 3 stopped
>1187992774 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 1:gfs1 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992774 1:gfs1 mark node 4 stopped
>1187992774 1:gfs1 process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992774 1:gfs1 action for app: start gfs1 39 2 4 2 4 3 5
>1187992774 got client 4 start_done
>1187992774 1:gfs1 send started
>1187992774 1:gfs1 mark node 4 started
>1187992774 1:gfs1 mark node 3 started
>1187992774 1:gfs1 mark node 5 started
>1187992774 1:gfs1 mark node 2 started
>1187992774 1:gfs1 process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992774 1:gfs1 action for app: finish gfs1 39
>1187992775 2:gfs2 confchg left 0 joined 1 total 4
>1187992775 2:gfs2 process_node_join 2
>1187992775 2:gfs2 cpg add node 2 total 4
>1187992775 2:gfs2 make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992775 2:gfs2 queue join event for nodeid 2
>1187992775 2:gfs2 process_current_event 200040001 2 JOIN_BEGIN
>1187992775 2:gfs2 action for app: stop gfs2
>1187992775 2:gfs2 app node join: add 2 total 4
>1187992775 got client 5 stop_done
>1187992775 2:gfs2 send stopped
>1187992775 2:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 2:gfs2 mark node 5 stopped
>1187992775 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 2:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 2:gfs2 mark node 3 stopped
>1187992775 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 2:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 2:gfs2 mark node 4 stopped
>1187992775 2:gfs2 process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992775 2:gfs2 action for app: start gfs2 40 2 4 2 4 3 5
>1187992775 got client 5 start_done
>1187992775 2:gfs2 send started
>1187992775 2:gfs2 mark node 3 started
>1187992775 2:gfs2 mark node 4 started
>1187992775 2:gfs2 mark node 5 started
>1187992775 2:gfs2 mark node 2 started
>1187992775 2:gfs2 process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992775 2:gfs2 action for app: finish gfs2 40
>1187992775 1:gfs2 confchg left 0 joined 1 total 4
>1187992775 1:gfs2 process_node_join 2
>1187992775 1:gfs2 cpg add node 2 total 4
>1187992775 1:gfs2 make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187992775 1:gfs2 queue join event for nodeid 2
>1187992775 1:gfs2 process_current_event 200040001 2 JOIN_BEGIN
>1187992775 1:gfs2 action for app: stop gfs2
>1187992775 1:gfs2 app node join: add 2 total 4
>1187992775 got client 4 stop_done
>1187992775 1:gfs2 send stopped
>1187992775 1:gfs2 waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 1:gfs2 mark node 5 stopped
>1187992775 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 1:gfs2 waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 1:gfs2 mark node 3 stopped
>1187992775 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 1:gfs2 waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187992775 1:gfs2 mark node 4 stopped
>1187992775 1:gfs2 process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187992775 1:gfs2 action for app: start gfs2 41 2 4 2 4 3 5
>1187992775 got client 4 start_done
>1187992775 1:gfs2 send started
>1187992775 1:gfs2 mark node 3 started
>1187992775 1:gfs2 mark node 4 started
>1187992775 1:gfs2 mark node 5 started
>1187992775 1:gfs2 mark node 2 started
>1187992775 1:gfs2 process_current_event 200040001 2 JOIN_ALL_STARTED
>1187992775 1:gfs2 action for app: finish gfs2 41
>1187992876 cman: lost quorum
>1187992876 cman: node 2 removed
>1187992876 add_recovery_set_cman nodeid 2
>1187992876 cman: node 3 removed
>1187992876 add_recovery_set_cman nodeid 3
>1187992876 cman: node 4 removed
>1187992876 add_recovery_set_cman nodeid 4
>1187992876 groupd confchg total 1 left 3 joined 0
>1187992876 add_recovery_set_cpg nodeid 2 procdown 0
>1187992876 0:default add to recovery set 2
>1187992876 1:clvmd add to recovery set 2
>1187992876 2:gfs1 add to recovery set 2
>1187992876 1:gfs1 add to recovery set 2
>1187992876 2:gfs2 add to recovery set 2
>1187992876 1:gfs2 add to recovery set 2
>1187992876 add_recovery_set_cpg nodeid 3 procdown 0
>1187992876 0:default add to recovery set 3
>1187992876 1:clvmd add to recovery set 3
>1187992876 2:gfs1 add to recovery set 3
>1187992876 1:gfs1 add to recovery set 3
>1187992876 2:gfs2 add to recovery set 3
>1187992876 1:gfs2 add to recovery set 3
>1187992876 add_recovery_set_cpg nodeid 4 procdown 0
>1187992876 0:default add to recovery set 4
>1187992876 1:clvmd add to recovery set 4
>1187992876 2:gfs1 add to recovery set 4
>1187992876 1:gfs1 add to recovery set 4
>1187992876 2:gfs2 add to recovery set 4
>1187992876 1:gfs2 add to recovery set 4
>1187992876 0:default confchg left 3 joined 0 total 1
>1187992876 0:default confchg removed node 2 reason 3
>1187992876 0:default process_node_down 2
>1187992876 0:default cpg del node 2 total 3 - down
>1187992876 0:default make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 0:default queue recover event for nodeid 2
>1187992876 0:default confchg removed node 3 reason 3
>1187992876 0:default process_node_down 3
>1187992876 0:default cpg del node 3 total 2 - down
>1187992876 0:default extend_recover_event for 2 with node 3
>1187992876 0:default confchg removed node 4 reason 3
>1187992876 0:default process_node_down 4
>1187992876 0:default cpg del node 4 total 1 - down
>1187992876 0:default extend_recover_event for 2 with node 4
>1187992876 1:clvmd confchg left 3 joined 0 total 1
>1187992876 1:clvmd confchg removed node 2 reason 3
>1187992876 1:clvmd process_node_down 2
>1187992876 1:clvmd cpg del node 2 total 3 - down
>1187992876 1:clvmd make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 1:clvmd queue recover event for nodeid 2
>1187992876 1:clvmd confchg removed node 3 reason 3
>1187992876 1:clvmd process_node_down 3
>1187992876 1:clvmd cpg del node 3 total 2 - down
>1187992876 1:clvmd extend_recover_event for 2 with node 3
>1187992876 1:clvmd confchg removed node 4 reason 3
>1187992876 1:clvmd process_node_down 4
>1187992876 1:clvmd cpg del node 4 total 1 - down
>1187992876 1:clvmd extend_recover_event for 2 with node 4
>1187992876 2:gfs1 confchg left 3 joined 0 total 1
>1187992876 2:gfs1 confchg removed node 2 reason 3
>1187992876 2:gfs1 process_node_down 2
>1187992876 2:gfs1 cpg del node 2 total 3 - down
>1187992876 2:gfs1 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 2:gfs1 queue recover event for nodeid 2
>1187992876 2:gfs1 confchg removed node 3 reason 3
>1187992876 2:gfs1 process_node_down 3
>1187992876 2:gfs1 cpg del node 3 total 2 - down
>1187992876 2:gfs1 extend_recover_event for 2 with node 3
>1187992876 2:gfs1 confchg removed node 4 reason 3
>1187992876 2:gfs1 process_node_down 4
>1187992876 2:gfs1 cpg del node 4 total 1 - down
>1187992876 2:gfs1 extend_recover_event for 2 with node 4
>1187992876 1:gfs1 confchg left 3 joined 0 total 1
>1187992876 1:gfs1 confchg removed node 2 reason 3
>1187992876 1:gfs1 process_node_down 2
>1187992876 1:gfs1 cpg del node 2 total 3 - down
>1187992876 1:gfs1 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 1:gfs1 queue recover event for nodeid 2
>1187992876 1:gfs1 confchg removed node 3 reason 3
>1187992876 1:gfs1 process_node_down 3
>1187992876 1:gfs1 cpg del node 3 total 2 - down
>1187992876 1:gfs1 extend_recover_event for 2 with node 3
>1187992876 1:gfs1 confchg removed node 4 reason 3
>1187992876 1:gfs1 process_node_down 4
>1187992876 1:gfs1 cpg del node 4 total 1 - down
>1187992876 1:gfs1 extend_recover_event for 2 with node 4
>1187992876 2:gfs2 confchg left 3 joined 0 total 1
>1187992876 2:gfs2 confchg removed node 2 reason 3
>1187992876 2:gfs2 process_node_down 2
>1187992876 2:gfs2 cpg del node 2 total 3 - down
>1187992876 2:gfs2 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 2:gfs2 queue recover event for nodeid 2
>1187992876 2:gfs2 confchg removed node 3 reason 3
>1187992876 2:gfs2 process_node_down 3
>1187992876 2:gfs2 cpg del node 3 total 2 - down
>1187992876 2:gfs2 extend_recover_event for 2 with node 3
>1187992876 2:gfs2 confchg removed node 4 reason 3
>1187992876 2:gfs2 process_node_down 4
>1187992876 2:gfs2 cpg del node 4 total 1 - down
>1187992876 2:gfs2 extend_recover_event for 2 with node 4
>1187992876 1:gfs2 confchg left 3 joined 0 total 1
>1187992876 1:gfs2 confchg removed node 2 reason 3
>1187992876 1:gfs2 process_node_down 2
>1187992876 1:gfs2 cpg del node 2 total 3 - down
>1187992876 1:gfs2 make_event_id 200030003 nodeid 2 memb_count 3 type 3
>1187992876 1:gfs2 queue recover event for nodeid 2
>1187992876 1:gfs2 confchg removed node 3 reason 3
>1187992876 1:gfs2 process_node_down 3
>1187992876 1:gfs2 cpg del node 3 total 2 - down
>1187992876 1:gfs2 extend_recover_event for 2 with node 3
>1187992876 1:gfs2 confchg removed node 4 reason 3
>1187992876 1:gfs2 process_node_down 4
>1187992876 1:gfs2 cpg del node 4 total 1 - down
>1187992876 1:gfs2 extend_recover_event for 2 with node 4
>1187992876 0:default set current event to recovery for 2
>1187992876 0:default process_current_event 200030003 2 FAIL_BEGIN
>1187992876 0:default action for app: stop default
>1187992876 1:clvmd set current event to recovery for 2
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_BEGIN
>1187992876 1:clvmd action for app: stop clvmd
>1187992876 2:gfs1 set current event to recovery for 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_BEGIN
>1187992876 2:gfs1 action for app: stop gfs1
>1187992876 1:gfs1 set current event to recovery for 2
>1187992876 1:gfs1 process_current_event 200030003 2 FAIL_BEGIN
>1187992876 1:gfs1 action for app: stop gfs1
>1187992876 2:gfs2 set current event to recovery for 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_BEGIN
>1187992876 2:gfs2 action for app: stop gfs2
>1187992876 1:gfs2 set current event to recovery for 2
>1187992876 1:gfs2 process_current_event 200030003 2 FAIL_BEGIN
>1187992876 1:gfs2 action for app: stop gfs2
>1187992876 got client 5 stop_done
>1187992876 2:gfs1 send stopped
>1187992876 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 got client 3 stop_done
>1187992876 0:default send stopped
>1187992876 got client 5 stop_done
>1187992876 2:gfs2 send stopped
>1187992876 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 mark node 5 stopped
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default mark node 5 stopped
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 mark node 5 stopped
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 got client 4 stop_done
>1187992876 1:clvmd send stopped
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 got client 4 stop_done
>1187992876 1:gfs1 send stopped
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 got client 4 stop_done
>1187992876 1:gfs2 send stopped
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd mark node 5 stopped
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 mark node 5 stopped
>1187992876 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 waiting for 1 more stopped messages before FAIL_ALL_STOPPED 2
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 mark node 5 stopped
>1187992876 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187992876 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 cman: have quorum
>1187993033 cman: node 2 added
>1187993033 cman: node 3 added
>1187993033 cman: node 4 added
>1187993033 0:default process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 0:default app node fail: del node 2 total 3
>1187993033 0:default app node fail: del node 4 total 2, ext
>1187993033 0:default app node fail: del node 3 total 1, ext
>1187993033 0:default action for app: start default 42 1 1 5
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 groupd confchg total 2 left 0 joined 1
>1187993033 joined node 2 had old max gid 50002
>1187993033 0:groupd cpg_mcast_joined retried 1
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 groupd confchg total 3 left 0 joined 1
>1187993033 0:default confchg left 0 joined 1 total 2
>1187993033 0:default process_node_join 2
>1187993033 0:default cpg add node 2 total 2
>1187993033 0:default queue_app_join: current event 2 200030003 FAIL_START_WAIT
>1187993033 0:default make_event_id 200020001 nodeid 2 memb_count 2 type 1
>1187993033 0:default queue join event for nodeid 2
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 groupd confchg total 4 left 0 joined 1
>1187993033 0:default confchg left 0 joined 1 total 3
>1187993033 0:default process_node_join 3
>1187993033 0:default cpg add node 3 total 3
>1187993033 0:default make_event_id 300030001 nodeid 3 memb_count 3 type 1
>1187993033 0:default queue join event for nodeid 3
>1187993033 0:default     queued ev 2 200020001 JOIN_BEGIN
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default confchg left 0 joined 1 total 4
>1187993033 0:default process_node_join 4
>1187993033 0:default cpg add node 4 total 4
>1187993033 0:default make_event_id 400040001 nodeid 4 memb_count 4 type 1
>1187993033 0:default queue join event for nodeid 4
>1187993033 0:default     queued ev 2 200020001 JOIN_BEGIN
>1187993033 0:default     queued ev 3 300030001 JOIN_BEGIN
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 300010001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 200020001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 200020001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 2 id 200020001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 2 id 400030001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 400030001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 2 id 400030001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 400030001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 4 id 400030001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 2 id 500040001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 500040001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 4 id 500040001 stopped
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 4 id 500040001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 2 id 500040001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993033 0:default ignore msg from 3 id 500040001 started
>1187993033 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:clvmd lower levels not recovered in rs 2
>1187993033 1:clvmd wait for lower_levels_recovered
>1187993033 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs1 lower levels not recovered in rs 2
>1187993033 2:gfs1 wait for lower_levels_recovered
>1187993033 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs1 lower levels not recovered in rs 2
>1187993033 1:gfs1 wait for lower_levels_recovered
>1187993033 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 2:gfs2 lower levels not recovered in rs 2
>1187993033 2:gfs2 wait for lower_levels_recovered
>1187993033 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993033 1:gfs2 lower levels not recovered in rs 2
>1187993033 1:gfs2 wait for lower_levels_recovered
>1187993034 client connection 12
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 got client 12 get_group
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 client connection 12
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 got client 12 get_group
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 client connection 12
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 got client 12 get_group
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 got client 3 start_done
>1187993034 0:default send started
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd lower levels not recovered in rs 2
>1187993034 1:clvmd wait for lower_levels_recovered
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 lower levels not recovered in rs 2
>1187993034 1:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 lower levels not recovered in rs 2
>1187993034 1:gfs2 wait for lower_levels_recovered
>1187993034 0:default mark node 5 started
>1187993034 0:default process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993034 0:default action for app: finish default 42
>1187993034 0:default rev 200030003 done, remove group from rs 2
>1187993034 0:default done in recovery set 2
>1187993034 recovery set 2 has 5 entries not done
>1187993034 0:default rev 200030003 done, remove group from rs 4
>1187993034 0:default done in recovery set 4
>1187993034 recovery set 4 has 5 entries not done
>1187993034 0:default rev 200030003 done, remove group from rs 3
>1187993034 0:default done in recovery set 3
>1187993034 recovery set 3 has 5 entries not done
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:clvmd app node fail: del node 2 total 3
>1187993034 1:clvmd app node fail: del node 4 total 2, ext
>1187993034 1:clvmd app node fail: del node 3 total 1, ext
>1187993034 1:clvmd action for app: start clvmd 43 1 1 5
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs1 app node fail: del node 2 total 3
>1187993034 1:gfs1 app node fail: del node 4 total 2, ext
>1187993034 1:gfs1 app node fail: del node 3 total 1, ext
>1187993034 1:gfs1 action for app: start gfs1 45 1 1 5
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 1:gfs2 app node fail: del node 2 total 3
>1187993034 1:gfs2 app node fail: del node 4 total 2, ext
>1187993034 1:gfs2 app node fail: del node 3 total 1, ext
>1187993034 1:gfs2 action for app: start gfs2 47 1 1 5
>1187993034 0:default process_current_event 200020001 2 JOIN_BEGIN
>1187993034 0:default action for app: stop default
>1187993034 0:default app node join: add 2 total 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 got client 3 stop_done
>1187993034 0:default send stopped
>1187993034 0:default mark_node_stopped: no nodeid 3
>1187993034 0:default mark_node_started: event not starting 2 from 3
>1187993034 0:default mark_node_started: no nodeid 3
>1187993034 0:default mark_node_started: event not starting 2 from 2
>1187993034 0:default mark node 2 started
>1187993034 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 got client 4 start_done
>1187993034 1:clvmd send started
>1187993034 0:default mark node 5 stopped
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 got client 4 start_done
>1187993034 1:gfs1 send started
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 got client 4 start_done
>1187993034 1:gfs2 send started
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 1:clvmd mark node 5 started
>1187993034 1:clvmd process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993034 1:clvmd action for app: finish clvmd 43
>1187993034 1:clvmd rev 200030003 done, remove group from rs 2
>1187993034 1:clvmd done in recovery set 2
>1187993034 recovery set 2 has 4 entries not done
>1187993034 1:clvmd rev 200030003 done, remove group from rs 4
>1187993034 1:clvmd done in recovery set 4
>1187993034 recovery set 4 has 4 entries not done
>1187993034 1:clvmd rev 200030003 done, remove group from rs 3
>1187993034 1:clvmd done in recovery set 3
>1187993034 recovery set 3 has 4 entries not done
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 1:gfs1 mark node 5 started
>1187993034 1:gfs1 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993034 1:gfs1 action for app: finish gfs1 45
>1187993034 1:gfs1 rev 200030003 done, remove group from rs 2
>1187993034 1:gfs1 done in recovery set 2
>1187993034 recovery set 2 has 3 entries not done
>1187993034 1:gfs1 rev 200030003 done, remove group from rs 4
>1187993034 1:gfs1 done in recovery set 4
>1187993034 recovery set 4 has 3 entries not done
>1187993034 1:gfs1 rev 200030003 done, remove group from rs 3
>1187993034 1:gfs1 done in recovery set 3
>1187993034 recovery set 3 has 3 entries not done
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 lower levels not recovered in rs 2
>1187993034 2:gfs1 wait for lower_levels_recovered
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 lower levels not recovered in rs 2
>1187993034 2:gfs2 wait for lower_levels_recovered
>1187993034 1:gfs2 mark node 5 started
>1187993034 1:gfs2 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993034 1:gfs2 action for app: finish gfs2 47
>1187993034 1:gfs2 rev 200030003 done, remove group from rs 2
>1187993034 1:gfs2 done in recovery set 2
>1187993034 recovery set 2 has 2 entries not done
>1187993034 1:gfs2 rev 200030003 done, remove group from rs 4
>1187993034 1:gfs2 done in recovery set 4
>1187993034 recovery set 4 has 2 entries not done
>1187993034 1:gfs2 rev 200030003 done, remove group from rs 3
>1187993034 1:gfs2 done in recovery set 3
>1187993034 recovery set 3 has 2 entries not done
>1187993034 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993034 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs1 app node fail: del node 2 total 3
>1187993034 2:gfs1 app node fail: del node 4 total 2, ext
>1187993034 2:gfs1 app node fail: del node 3 total 1, ext
>1187993034 2:gfs1 action for app: start gfs1 44 1 1 5
>1187993034 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STOPPED
>1187993034 2:gfs2 app node fail: del node 2 total 3
>1187993034 2:gfs2 app node fail: del node 4 total 2, ext
>1187993034 2:gfs2 app node fail: del node 3 total 1, ext
>1187993034 2:gfs2 action for app: start gfs2 46 1 1 5
>1187993035 got client 5 start_done
>1187993035 2:gfs2 send started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 2:gfs2 mark node 5 started
>1187993035 2:gfs2 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993035 2:gfs2 action for app: finish gfs2 46
>1187993035 2:gfs2 rev 200030003 done, remove group from rs 2
>1187993035 2:gfs2 done in recovery set 2
>1187993035 recovery set 2 has 1 entries not done
>1187993035 2:gfs2 rev 200030003 done, remove group from rs 4
>1187993035 2:gfs2 done in recovery set 4
>1187993035 recovery set 4 has 1 entries not done
>1187993035 2:gfs2 rev 200030003 done, remove group from rs 3
>1187993035 2:gfs2 done in recovery set 3
>1187993035 recovery set 3 has 1 entries not done
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 got client 5 start_done
>1187993035 2:gfs1 send started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 2:gfs1 mark node 5 started
>1187993035 2:gfs1 process_current_event 200030003 2 FAIL_ALL_STARTED
>1187993035 2:gfs1 action for app: finish gfs1 44
>1187993035 2:gfs1 rev 200030003 done, remove group from rs 2
>1187993035 2:gfs1 done in recovery set 2
>1187993035 recovery set 2 is all done
>1187993035 2:gfs1 rev 200030003 done, remove group from rs 4
>1187993035 2:gfs1 done in recovery set 4
>1187993035 recovery set 4 is all done
>1187993035 2:gfs1 rev 200030003 done, remove group from rs 3
>1187993035 2:gfs1 done in recovery set 3
>1187993035 recovery set 3 is all done
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd confchg left 0 joined 1 total 2
>1187993035 1:clvmd process_node_join 4
>1187993035 1:clvmd cpg add node 4 total 2
>1187993035 1:clvmd make_event_id 400020001 nodeid 4 memb_count 2 type 1
>1187993035 1:clvmd queue join event for nodeid 4
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd process_current_event 400020001 4 JOIN_BEGIN
>1187993035 1:clvmd action for app: stop clvmd
>1187993035 1:clvmd app node join: add 4 total 2
>1187993035 got client 4 stop_done
>1187993035 1:clvmd send stopped
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 4
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 5 stopped
>1187993035 1:clvmd process_current_event 400020001 4 JOIN_ALL_STOPPED
>1187993035 1:clvmd action for app: start clvmd 51 2 2 4 5
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 got client 4 start_done
>1187993035 1:clvmd send started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 5 started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 4 started
>1187993035 1:clvmd process_current_event 400020001 4 JOIN_ALL_STARTED
>1187993035 1:clvmd action for app: finish clvmd 51
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd confchg left 0 joined 1 total 3
>1187993035 1:clvmd process_node_join 3
>1187993035 1:clvmd cpg add node 3 total 3
>1187993035 1:clvmd make_event_id 300030001 nodeid 3 memb_count 3 type 1
>1187993035 1:clvmd queue join event for nodeid 3
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd process_current_event 300030001 3 JOIN_BEGIN
>1187993035 1:clvmd action for app: stop clvmd
>1187993035 1:clvmd app node join: add 3 total 3
>1187993035 got client 4 stop_done
>1187993035 1:clvmd send stopped
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 3
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 4 stopped
>1187993035 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 3
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 5 stopped
>1187993035 1:clvmd process_current_event 300030001 3 JOIN_ALL_STOPPED
>1187993035 1:clvmd action for app: start clvmd 52 2 3 3 4 5
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 got client 4 start_done
>1187993035 1:clvmd send started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 4 started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 5 started
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993035 1:clvmd mark node 3 started
>1187993035 1:clvmd process_current_event 300030001 3 JOIN_ALL_STARTED
>1187993035 1:clvmd action for app: finish clvmd 52
>1187993035 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd confchg left 0 joined 1 total 4
>1187993036 1:clvmd process_node_join 2
>1187993036 1:clvmd cpg add node 2 total 4
>1187993036 1:clvmd make_event_id 200040001 nodeid 2 memb_count 4 type 1
>1187993036 1:clvmd queue join event for nodeid 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd process_current_event 200040001 2 JOIN_BEGIN
>1187993036 1:clvmd action for app: stop clvmd
>1187993036 1:clvmd app node join: add 2 total 4
>1187993036 got client 4 stop_done
>1187993036 1:clvmd send stopped
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd waiting for 3 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 5 stopped
>1187993036 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd waiting for 2 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 3 stopped
>1187993036 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 4 stopped
>1187993036 1:clvmd process_current_event 200040001 2 JOIN_ALL_STOPPED
>1187993036 1:clvmd action for app: start clvmd 53 2 4 2 3 4 5
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 got client 4 start_done
>1187993036 1:clvmd send started
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 4 started
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 5 started
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 2 started
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993036 1:clvmd mark node 3 started
>1187993036 1:clvmd process_current_event 200040001 2 JOIN_ALL_STARTED
>1187993036 1:clvmd action for app: finish clvmd 53
>1187993036 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993053 client connection 12
>1187993053 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993053 got client 12 get_groups
>1187993053 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993053 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993057 client connection 12
>1187993057 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993057 got client 12 get_groups
>1187993057 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993057 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993061 client connection 12
>1187993061 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993061 got client 12 get_groups
>1187993061 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993061 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993065 client connection 12
>1187993065 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993065 got client 12 get_groups
>1187993065 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993065 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993069 client connection 12
>1187993069 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993069 got client 12 get_groups
>1187993069 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993069 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993073 client connection 12
>1187993073 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993073 got client 12 get_groups
>1187993073 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993073 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993077 client connection 12
>1187993077 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993077 got client 12 get_groups
>1187993077 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993077 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993081 client connection 12
>1187993081 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993081 got client 12 get_groups
>1187993081 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993081 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993085 client connection 12
>1187993085 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993085 got client 12 get_groups
>1187993085 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993085 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993090 client connection 12
>1187993090 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993090 got client 12 get_groups
>1187993090 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993090 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993094 client connection 12
>1187993094 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993094 got client 12 get_groups
>1187993094 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993094 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993098 client connection 12
>1187993098 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993098 got client 12 get_groups
>1187993098 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993098 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993102 client connection 12
>1187993102 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993102 got client 12 get_groups
>1187993102 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993102 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993106 client connection 12
>1187993106 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993106 got client 12 get_groups
>1187993106 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993106 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993110 client connection 12
>1187993110 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993110 got client 12 get_groups
>1187993110 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993110 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993114 client connection 12
>1187993114 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993114 got client 12 get_groups
>1187993114 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993114 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993118 client connection 12
>1187993118 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993118 got client 12 get_groups
>1187993118 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993118 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993122 client connection 12
>1187993122 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993122 got client 12 get_groups
>1187993122 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993122 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993126 client connection 12
>1187993126 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993126 got client 12 get_groups
>1187993126 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993126 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993130 client connection 12
>1187993130 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993130 got client 12 get_groups
>1187993130 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993130 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993134 client connection 12
>1187993134 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993134 got client 12 get_groups
>1187993134 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993134 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993138 client connection 12
>1187993138 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993138 got client 12 get_groups
>1187993138 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993138 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993142 client connection 12
>1187993142 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993142 got client 12 get_groups
>1187993142 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993142 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993146 client connection 12
>1187993146 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993146 got client 12 get_groups
>1187993146 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993146 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993150 client connection 12
>1187993150 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993150 got client 12 get_groups
>1187993150 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993150 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993154 client connection 12
>1187993154 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993154 got client 12 get_groups
>1187993154 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1187993154 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188222514 client connection 12
>1188222514 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188222514 got client 12 get_groups
>1188222514 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188222514 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226261 client connection 12
>1188226261 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226261 got client 12 get_groups
>1188226261 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226261 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226644 client connection 12
>1188226644 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226644 got client 12 dump
>1188226644 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226645 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226681 client connection 12
>1188226681 0:default waiting for 1 more stopped messages before JOIN_ALL_STOPPED 2
>1188226681 got client 12 dump
>

Actions: View

Attachments on bug 251966: 161197 | 161199 | 161201 | 161864 | 173681 | 173701 | 173721 | 173741 | 173901 | 173961