Login
Log in using an SSO provider:
Fedora Account System
Red Hat Associate
Red Hat Customer
Login using a Red Hat Bugzilla account
Forgot Password
Create an Account
Red Hat Bugzilla – Attachment 1981342 Details for
Bug 2224249
Manual and Automatic failover issues in RHEL 9
Home
New
Search
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh89 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
[?]
This site requires JavaScript to be enabled to function correctly, please enable it.
LatestLogs_post_call_with_MS&Rhel
pacemaker_rhel92.log (text/plain), 324.99 KB, created by
Aravind Mahadevan
on 2023-08-02 18:23:27 UTC
(
hide
)
Description:
LatestLogs_post_call_with_MS&Rhel
Filename:
MIME Type:
Creator:
Aravind Mahadevan
Created:
2023-08-02 18:23:27 UTC
Size:
324.99 KB
patch
obsolete
>Aug 01 10:59:21.106 rhel92 pacemakerd [1478932] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.106 rhel92 pacemakerd [1478932] (ipc_post_disconnect) info: Disconnected from launcher IPC API >Aug 01 10:59:21.109 rhel92 pacemakerd [1478932] (get_cluster_type) info: Detected an active 'corosync' cluster >Aug 01 10:59:21.109 rhel92 pacemakerd [1478932] (mcp_read_config) info: Reading configuration for corosync stack >Aug 01 10:59:21.109 rhel92 pacemakerd [1478932] (main) notice: Starting Pacemaker 2.1.5-9.el9_2 | build=a3f44794f94 features:agent-manpages cibsecrets corosync-ge-2 default-concurrent-fencing default-resource-stickiness default-sbd-sync generated-manpages monotonic nagios ncurses remote systemd >Aug 01 10:59:21.109 rhel92 pacemakerd [1478932] (qb_ipcs_us_publish) info: server name: pacemakerd >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to cib_ro IPC: Connection refused >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to stonith-ng IPC: Connection refused >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to lrmd IPC: Connection refused >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection refused >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to pengine IPC: Connection refused >Aug 01 10:59:21.116 rhel92 pacemakerd [1478932] (pcmk__ipc_is_authentic_process_active) info: Could not connect to crmd IPC: Connection refused >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Using uid=189 and group=189 for process pacemaker-based >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478933 for process pacemaker-based >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478934 for process pacemaker-fenced >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478935 for process pacemaker-execd >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Using uid=189 and group=189 for process pacemaker-attrd >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478936 for process pacemaker-attrd >Aug 01 10:59:21.117 rhel92 pacemakerd [1478932] (start_child) info: Using uid=189 and group=189 for process pacemaker-schedulerd >Aug 01 10:59:21.118 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478937 for process pacemaker-schedulerd >Aug 01 10:59:21.118 rhel92 pacemakerd [1478932] (start_child) info: Using uid=189 and group=189 for process pacemaker-controld >Aug 01 10:59:21.118 rhel92 pacemakerd [1478932] (start_child) info: Forked child 1478938 for process pacemaker-controld >Aug 01 10:59:21.118 rhel92 pacemakerd [1478932] (main) notice: Pacemaker daemon successfully started and accepting connections >Aug 01 10:59:21.120 rhel92 pacemaker-execd [1478935] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.120 rhel92 pacemaker-execd [1478935] (main) notice: Starting Pacemaker local executor >Aug 01 10:59:21.120 rhel92 pacemaker-execd [1478935] (qb_ipcs_us_publish) info: server name: lrmd >Aug 01 10:59:21.120 rhel92 pacemaker-execd [1478935] (main) notice: Pacemaker local executor successfully started and accepting connections >Aug 01 10:59:21.120 rhel92 pacemaker-execd [1478935] (main) notice: OCF resource agent search path is /usr/lib/ocf/resource.d >Aug 01 10:59:21.120 rhel92 pacemaker-fenced [1478934] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.120 rhel92 pacemaker-fenced [1478934] (main) notice: Starting Pacemaker fencer >Aug 01 10:59:21.120 rhel92 pacemaker-fenced [1478934] (get_cluster_type) info: Verifying cluster type: 'corosync' >Aug 01 10:59:21.120 rhel92 pacemaker-fenced [1478934] (get_cluster_type) info: Assuming an active 'corosync' cluster >Aug 01 10:59:21.120 rhel92 pacemaker-fenced [1478934] (crm_cluster_connect) notice: Connecting to corosync cluster infrastructure >Aug 01 10:59:21.121 rhel92 pacemaker-based [1478933] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.121 rhel92 pacemaker-based [1478933] (main) notice: Starting Pacemaker CIB manager >Aug 01 10:59:21.122 rhel92 pacemaker-schedulerd[1478937] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.122 rhel92 pacemaker-schedulerd[1478937] (main) notice: Starting Pacemaker scheduler >Aug 01 10:59:21.122 rhel92 pacemaker-schedulerd[1478937] (qb_ipcs_us_publish) info: server name: pengine >Aug 01 10:59:21.122 rhel92 pacemaker-attrd [1478936] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.122 rhel92 pacemaker-schedulerd[1478937] (main) notice: Pacemaker scheduler successfully started and accepting connections >Aug 01 10:59:21.122 rhel92 pacemaker-attrd [1478936] (main) notice: Starting Pacemaker node attribute manager >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (get_cluster_type) info: Verifying cluster type: 'corosync' >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (get_cluster_type) info: Assuming an active 'corosync' cluster >Aug 01 10:59:21.122 rhel92 pacemaker-attrd [1478936] (ipc_post_disconnect) info: Disconnected from attribute manager IPC API >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (pcmk__daemon_can_write) notice: /var/lib/pacemaker/cib/cib.xml not found: No such file or directory >Aug 01 10:59:21.122 rhel92 pacemaker-attrd [1478936] (cib_native_signon_raw) info: Could not connect to CIB manager for attrd >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (pcmk__daemon_can_write) notice: /var/lib/pacemaker/cib/cib.xml.sig not found: No such file or directory >Aug 01 10:59:21.122 rhel92 pacemaker-attrd [1478936] (cib_native_signon_raw) info: Connection to CIB manager for attrd failed: Transport endpoint is not connected >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (retrieveCib) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig) >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (cib_file_read_and_verify) warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2) >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (readCibXmlFile) warning: Primary configuration corrupt or unusable, trying backups in /var/lib/pacemaker/cib >Aug 01 10:59:21.122 rhel92 pacemaker-based [1478933] (readCibXmlFile) warning: Continuing with an empty configuration. >Aug 01 10:59:21.123 rhel92 pacemaker-controld [1478938] (crm_log_init) info: Changed active directory to /var/lib/pacemaker/cores >Aug 01 10:59:21.123 rhel92 pacemaker-controld [1478938] (main) notice: Starting Pacemaker controller >Aug 01 10:59:21.123 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_STARTUP received in state S_STARTING from crmd_init >Aug 01 10:59:21.123 rhel92 pacemaker-controld [1478938] (cib_native_signon_raw) info: Could not connect to CIB manager for crmd >Aug 01 10:59:21.123 rhel92 pacemaker-controld [1478938] (cib_native_signon_raw) info: Connection to CIB manager for crmd failed: Transport endpoint is not connected >Aug 01 10:59:21.124 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Created entry 95a04059-ac2b-4a0f-971e-6cb82fa9ec90/0x559d04da6ef0 for node rhel92/2 (1 total) >Aug 01 10:59:21.124 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 2 is now known as rhel92 >Aug 01 10:59:21.128 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 2 has uuid 2 >Aug 01 10:59:21.128 rhel92 pacemaker-fenced [1478934] (crm_update_peer_proc) info: cluster_connect_cpg: Node rhel92[2] - corosync-cpg is now online >Aug 01 10:59:21.128 rhel92 pacemaker-fenced [1478934] (update_peer_state_iter) notice: Node rhel92 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:21.128 rhel92 pacemaker-fenced [1478934] (pcmk__corosync_connect) info: Connection to corosync established >Aug 01 10:59:21.130 rhel92 pacemaker-fenced [1478934] (cib_native_signon_raw) info: Could not connect to CIB manager for stonithd >Aug 01 10:59:21.130 rhel92 pacemaker-fenced [1478934] (cib_native_signon_raw) info: Connection to CIB manager for stonithd failed: Transport endpoint is not connected >Aug 01 10:59:21.131 rhel92 pacemaker-based [1478933] (crm_cluster_connect) notice: Connecting to corosync cluster infrastructure >Aug 01 10:59:21.134 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Created entry b8843a66-89d7-467e-9b37-95ea9316362b/0x55c8f0f5feb0 for node rhel92/2 (1 total) >Aug 01 10:59:21.134 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 2 is now known as rhel92 >Aug 01 10:59:21.136 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 2 has uuid 2 >Aug 01 10:59:21.136 rhel92 pacemaker-based [1478933] (crm_update_peer_proc) info: cluster_connect_cpg: Node rhel92[2] - corosync-cpg is now online >Aug 01 10:59:21.136 rhel92 pacemaker-based [1478933] (update_peer_state_iter) notice: Node rhel92 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:21.137 rhel92 pacemaker-based [1478933] (pcmk__corosync_connect) info: Connection to corosync established >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (qb_ipcs_us_publish) info: server name: cib_ro >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (qb_ipcs_us_publish) info: server name: cib_rw >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (qb_ipcs_us_publish) info: server name: cib_shm >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (main) notice: Pacemaker CIB manager successfully started and accepting connections >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 0: node 2 pid 1478933 joined via cpg_join >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 0: rhel92 (node 2 pid 1478933) is member >Aug 01 10:59:21.139 rhel92 pacemaker-based [1478933] (cib_file_read_and_verify) warning: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2) >Aug 01 10:59:21.147 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw >Aug 01 10:59:21.156 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.0.0 of the CIB to disk (digest: d6b33ef8090fc572254da1fc49c0b24d) >Aug 01 10:59:21.164 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.j0g4GT (digest: /var/lib/pacemaker/cib/cib.ghh6QF) >Aug 01 10:59:22.123 rhel92 pacemaker-attrd [1478936] (cib_native_signon_raw) info: Successfully connected to CIB manager for attrd >Aug 01 10:59:22.123 rhel92 pacemaker-attrd [1478936] (main) info: CIB connection active >Aug 01 10:59:22.123 rhel92 pacemaker-attrd [1478936] (get_cluster_type) info: Verifying cluster type: 'corosync' >Aug 01 10:59:22.123 rhel92 pacemaker-attrd [1478936] (get_cluster_type) info: Assuming an active 'corosync' cluster >Aug 01 10:59:22.123 rhel92 pacemaker-attrd [1478936] (crm_cluster_connect) notice: Connecting to corosync cluster infrastructure >Aug 01 10:59:22.125 rhel92 pacemaker-controld [1478938] (cib_native_signon_raw) info: Successfully connected to CIB manager for crmd >Aug 01 10:59:22.125 rhel92 pacemaker-controld [1478938] (get_cluster_type) info: Verifying cluster type: 'corosync' >Aug 01 10:59:22.125 rhel92 pacemaker-controld [1478938] (get_cluster_type) info: Assuming an active 'corosync' cluster >Aug 01 10:59:22.125 rhel92 pacemaker-controld [1478938] (crm_cluster_connect) notice: Connecting to corosync cluster infrastructure >Aug 01 10:59:22.129 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Created entry fbc2c4b7-1688-4040-bcbc-e2ad6b5b7b6b/0x55ea58a8d150 for node rhel92/2 (1 total) >Aug 01 10:59:22.129 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 2 is now known as rhel92 >Aug 01 10:59:22.130 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Created entry 9dc1670b-80b3-45be-95ee-eaf34949dab9/0x560949754130 for node rhel92/2 (1 total) >Aug 01 10:59:22.130 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 2 is now known as rhel92 >Aug 01 10:59:22.130 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel92 is now in unknown state >Aug 01 10:59:22.131 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 2 has uuid 2 >Aug 01 10:59:22.131 rhel92 pacemaker-attrd [1478936] (crm_update_peer_proc) info: cluster_connect_cpg: Node rhel92[2] - corosync-cpg is now online >Aug 01 10:59:22.131 rhel92 pacemaker-attrd [1478936] (update_peer_state_iter) notice: Node rhel92 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:22.131 rhel92 pacemaker-attrd [1478936] (pcmk__corosync_connect) info: Connection to corosync established >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (cib_native_signon_raw) info: Successfully connected to CIB manager for stonithd >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (setup_cib) info: Watching for fencing topology changes >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (qb_ipcs_us_publish) info: server name: stonith-ng >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (main) notice: Pacemaker fencer successfully started and accepting connections >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (init_cib_cache_cb) info: Updating device list from CIB >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (cib_devices_update) info: Updating devices to version 0.0.0 >Aug 01 10:59:22.131 rhel92 pacemaker-fenced [1478934] (unpack_nodes) info: Creating a fake local node >Aug 01 10:59:22.132 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 0: node 2 pid 1478934 joined via cpg_join >Aug 01 10:59:22.132 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 0: rhel92 (node 2 pid 1478934) is member >Aug 01 10:59:22.133 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 2 has uuid 2 >Aug 01 10:59:22.133 rhel92 pacemaker-controld [1478938] (crm_update_peer_proc) info: cluster_connect_cpg: Node rhel92[2] - corosync-cpg is now online >Aug 01 10:59:22.135 rhel92 pacemaker-attrd [1478936] (main) info: Cluster connection active >Aug 01 10:59:22.135 rhel92 pacemaker-attrd [1478936] (attrd_erase_attrs) info: Clearing transient attributes from CIB | xpath=//node_state[@uname='rhel92']/transient_attributes >Aug 01 10:59:22.135 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='rhel92']/transient_attributes to all (origin=local/attrd/2) >Aug 01 10:59:22.138 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Node rhel92 is now a peer | DC=<none> old=0000000 new=0x4000000 >Aug 01 10:59:22.138 rhel92 pacemaker-controld [1478938] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds >Aug 01 10:59:22.138 rhel92 pacemaker-controld [1478938] (pcmk__corosync_connect) info: Connection to corosync established >Aug 01 10:59:22.139 rhel92 pacemaker-attrd [1478936] (qb_ipcs_us_publish) info: server name: attrd >Aug 01 10:59:22.139 rhel92 pacemaker-attrd [1478936] (main) notice: Pacemaker node attribute manager successfully started and accepting connections >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 0: node 2 pid 1478936 joined via cpg_join >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 0: rhel92 (node 2 pid 1478936) is member >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #attrd-protocol[rhel92]: (unset) -> 5 | from rhel92 with no write delay >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (attrd_start_election_if_needed) info: Starting an election to determine the writer >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (election_check) info: election-attrd won by local node >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset) >Aug 01 10:59:22.140 rhel92 pacemaker-attrd [1478936] (attrd_write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a >Aug 01 10:59:22.140 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rhel92']/transient_attributes: OK (rc=0, origin=rhel92/attrd/2, version=0.0.0) >Aug 01 10:59:22.142 rhel92 pacemaker-controld [1478938] (pcmk__corosync_quorum_connect) warning: No quorum >Aug 01 10:59:22.143 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Created entry a501b213-00fb-45b8-aa3b-361962bbafa5/0x560949757b50 for node rhel91/1 (2 total) >Aug 01 10:59:22.143 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 1 is now known as rhel91 >Aug 01 10:59:22.143 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel91 is now in unknown state >Aug 01 10:59:22.143 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 1 has uuid 1 >Aug 01 10:59:22.144 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Created entry 621d9f77-b63c-4a67-a903-7e678a52a0b5/0x560949757bd0 for node rhel93/3 (3 total) >Aug 01 10:59:22.144 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 3 is now known as rhel93 >Aug 01 10:59:22.144 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel93 is now in unknown state >Aug 01 10:59:22.144 rhel92 pacemaker-controld [1478938] (crm_get_peer) info: Node 3 has uuid 3 >Aug 01 10:59:22.146 rhel92 pacemaker-controld [1478938] (do_ha_control) info: Connected to the cluster >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/5) >Aug 01 10:59:22.146 rhel92 pacemaker-controld [1478938] (lrmd_ipc_connect) info: Connecting to executor >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.0.0 2 >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.1.0 (null) >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=1 >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/nodes: <node id="1" uname="rhel91"/> >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/nodes: <node id="2" uname="rhel92"/> >Aug 01 10:59:22.146 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/nodes: <node id="3" uname="rhel93"/> >Aug 01 10:59:22.147 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=rhel92/crmd/5, version=0.1.0) >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (try_local_executor_connect) info: Connection to the local executor established >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_started) info: Delaying start, no membership data (0000000000100000) >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_started) info: Delaying start, no membership data (0000000000100000) >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (quorum_notification_cb) info: Quorum still lost | membership=18 members=1 >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (update_peer_state_iter) notice: Node rhel92 state is now member | nodeid=2 previous=unknown source=quorum_notification_cb >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel92 is now member (was in unknown state) >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (pcmk__reap_unseen_nodes) info: State of node rhel91[1] is still unknown >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (pcmk__reap_unseen_nodes) info: State of node rhel93[3] is still unknown >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_started) info: Delaying start, Config not read (0000000000000040) >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (qb_ipcs_us_publish) info: server name: crmd >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_started) notice: Pacemaker controller successfully started and accepting connections >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_PENDING received in state S_STARTING from do_started >Aug 01 10:59:22.147 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_STARTING -> S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started >Aug 01 10:59:22.155 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw >Aug 01 10:59:22.164 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.1.0 of the CIB to disk (digest: f1e4544c7603311cff669b62d6728952) >Aug 01 10:59:22.171 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.DZzYrM (digest: /var/lib/pacemaker/cib/cib.fwglin) >Aug 01 10:59:23.148 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 0: node 2 pid 1478938 joined via cpg_join >Aug 01 10:59:23.148 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 0: rhel92 (node 2 pid 1478938) is member >Aug 01 10:59:23.149 rhel92 pacemaker-controld [1478938] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 30 seconds >Aug 01 10:59:23.149 rhel92 pacemaker-controld [1478938] (te_connect_stonith) notice: Fencer successfully connected >Aug 01 10:59:23.354 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 1: node 1 pid 3734571 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 1: node 3 pid 953381 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 1: node 1 pid 3734573 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 1: node 3 pid 953383 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 1: rhel91 (node 1 pid 3734573) is member >Aug 01 10:59:23.354 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 1 pid 3734569 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel91[1] - corosync-cpg is now online >Aug 01 10:59:23.354 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 1: node 3 pid 953379 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Node rhel91 is now a peer | DC=<none> old=0000000 new=0x4000000 >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 1: rhel92 (node 2 pid 1478938) is member >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (pcmk_cpg_membership) info: Group crmd event 1: rhel93 (node 3 pid 953383) is member >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel93[3] - corosync-cpg is now online >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Node rhel93 is now a peer | DC=<none> old=0000000 new=0x4000000 >Aug 01 10:59:23.354 rhel92 pacemaker-controld [1478938] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds >Aug 01 10:59:23.354 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 1: node 1 pid 3734568 joined via cluster join >Aug 01 10:59:23.354 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 1: node 3 pid 953378 joined via cluster join >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Created entry f1dca08a-6eb9-4042-b0d8-8b941dcfef11/0x55ea58a8f660 for node rhel91/1 (2 total) >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 1 is now known as rhel91 >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 1 has uuid 1 >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 1: rhel91 (node 1 pid 3734571) is member >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel91[1] - corosync-cpg is now online >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (update_peer_state_iter) notice: Node rhel91 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.359 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 1: rhel92 (node 2 pid 1478936) is member >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Created entry d034fbdd-5004-4ec7-875b-c5936832a9ed/0x559d04dbbb10 for node rhel91/1 (2 total) >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 1 is now known as rhel91 >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 1 has uuid 1 >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 1: rhel91 (node 1 pid 3734569) is member >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel91[1] - corosync-cpg is now online >Aug 01 10:59:23.359 rhel92 pacemaker-fenced [1478934] (update_peer_state_iter) notice: Node rhel91 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.360 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 1: rhel92 (node 2 pid 1478934) is member >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Created entry 4457086d-4a4a-46ad-9eb0-1425fefadb3f/0x55c8f0f62f00 for node rhel91/1 (2 total) >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 1 is now known as rhel91 >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 1 has uuid 1 >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 1: rhel91 (node 1 pid 3734568) is member >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel91[1] - corosync-cpg is now online >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (update_peer_state_iter) notice: Node rhel91 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.363 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 1: rhel92 (node 2 pid 1478933) is member >Aug 01 10:59:23.364 rhel92 pacemaker-controld [1478938] (quorum_notification_cb) notice: Quorum acquired | membership=26 members=3 >Aug 01 10:59:23.364 rhel92 pacemaker-controld [1478938] (update_peer_state_iter) notice: Node rhel91 state is now member | nodeid=1 previous=unknown source=quorum_notification_cb >Aug 01 10:59:23.365 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel91 is now member (was in unknown state) >Aug 01 10:59:23.365 rhel92 pacemaker-controld [1478938] (update_peer_state_iter) notice: Node rhel93 state is now member | nodeid=3 previous=unknown source=quorum_notification_cb >Aug 01 10:59:23.365 rhel92 pacemaker-controld [1478938] (peer_update_callback) info: Cluster node rhel93 is now member (was in unknown state) >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Created entry b7f054dc-2a3d-48be-8d90-511c6d583e46/0x55ea58a94300 for node rhel93/3 (3 total) >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 3 is now known as rhel93 >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (crm_get_peer) info: Node 3 has uuid 3 >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (pcmk_cpg_membership) info: Group attrd event 1: rhel93 (node 3 pid 953381) is member >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel93[3] - corosync-cpg is now online >Aug 01 10:59:23.365 rhel92 pacemaker-attrd [1478936] (update_peer_state_iter) notice: Node rhel93 state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Created entry c029d429-6a82-4a16-a1a6-3f3ec4692b4b/0x559d04dad7c0 for node rhel93/3 (3 total) >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 3 is now known as rhel93 >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (crm_get_peer) info: Node 3 has uuid 3 >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (pcmk_cpg_membership) info: Group stonith-ng event 1: rhel93 (node 3 pid 953379) is member >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Created entry 630f7a9a-eea7-491a-a5b1-4030c95fdf84/0x55c8f0ff1900 for node rhel93/3 (3 total) >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel93[3] - corosync-cpg is now online >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 3 is now known as rhel93 >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (crm_get_peer) info: Node 3 has uuid 3 >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (pcmk_cpg_membership) info: Group cib event 1: rhel93 (node 3 pid 953378) is member >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (crm_update_peer_proc) info: pcmk_cpg_membership: Node rhel93[3] - corosync-cpg is now online >Aug 01 10:59:23.365 rhel92 pacemaker-based [1478933] (update_peer_state_iter) notice: Node rhel93 state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.365 rhel92 pacemaker-fenced [1478934] (update_peer_state_iter) notice: Node rhel93 state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc >Aug 01 10:59:23.570 rhel92 pacemaker-attrd [1478936] (attrd_check_for_new_writer) notice: Detected another attribute writer (rhel93), starting new election >Aug 01 10:59:23.570 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel93 >Aug 01 10:59:23.570 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #attrd-protocol[rhel93]: (unset) -> 5 | from rhel93 with no write delay >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (attrd_check_for_new_writer) notice: Recorded new attribute writer: rhel91 (was rhel92) >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel91 >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #attrd-protocol[rhel91]: (unset) -> 5 | from rhel91 with no write delay >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 2 (owner node ID 1) lost: vote from rhel91 (Uptime) >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel91 >Aug 01 10:59:23.571 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel91 >Aug 01 10:59:23.572 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 2 (owner node ID 3) lost: vote from rhel93 (Uptime) >Aug 01 10:59:23.572 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel93 >Aug 01 10:59:23.572 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel93 >Aug 01 10:59:23.572 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 3 (owner node ID 3) lost: vote from rhel93 (Uptime) >Aug 01 10:59:23.572 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 3 (owner node ID 1) lost: vote from rhel91 (Uptime) >Aug 01 10:59:23.573 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 4 (owner node ID 3) lost: vote from rhel93 (Uptime) >Aug 01 10:59:23.573 rhel92 pacemaker-attrd [1478936] (election_count_vote) info: election-attrd round 5 (owner node ID 3) lost: vote from rhel93 (Uptime) >Aug 01 10:59:23.574 rhel92 pacemaker-attrd [1478936] (attrd_check_for_new_writer) notice: Recorded new attribute writer: rhel93 (was rhel91) >Aug 01 10:59:23.574 rhel92 pacemaker-attrd [1478936] (attrd_peer_sync_response) info: Processing sync-response from rhel93 >Aug 01 10:59:43.148 rhel92 pacemaker-controld [1478938] (crm_timer_popped) info: Election Trigger just popped | input=I_DC_TIMEOUT time=20000ms >Aug 01 10:59:43.148 rhel92 pacemaker-controld [1478938] (do_log) warning: Input I_DC_TIMEOUT received in state S_PENDING from crm_timer_popped >Aug 01 10:59:43.148 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_PENDING -> S_ELECTION | input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped >Aug 01 10:59:43.150 rhel92 pacemaker-controld [1478938] (election_check) info: election-DC won by local node >Aug 01 10:59:43.150 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb >Aug 01 10:59:43.150 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb >Aug 01 10:59:43.150 rhel92 pacemaker-controld [1478938] (do_te_control) info: Registering TE UUID: 976395e4-0f30-4f7e-9b5f-1da0b320d7c4 >Aug 01 10:59:43.161 rhel92 pacemaker-controld [1478938] (do_dc_takeover) info: Taking over DC status for this partition >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_readwrite) info: We are now in R/W mode >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/8, version=0.1.0) >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/9) >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=rhel92/crmd/9, version=0.1.0) >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/11) >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/13) >Aug 01 10:59:43.162 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/15) >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.1.0 2 >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.2.0 (null) >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=2 >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config: <cluster_property_set id="cib-bootstrap-options"/> >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/> >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </cluster_property_set> >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=rhel92/crmd/11, version=0.2.0) >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.2.0 2 >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.3.0 (null) >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=3 >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.5-9.el9_2-a3f44794f94"/> >Aug 01 10:59:43.163 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=rhel92/crmd/13, version=0.3.0) >Aug 01 10:59:43.164 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.3.0 2 >Aug 01 10:59:43.164 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.4.0 (null) >Aug 01 10:59:43.164 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=4 >Aug 01 10:59:43.164 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/> >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=rhel92/crmd/15, version=0.4.0) >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (join_make_offer) info: Making join-1 offers based on membership event 26 >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/17) >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (join_make_offer) info: Sending join-1 offer to rhel91 >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (join_make_offer) info: Sending join-1 offer to rhel92 >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (join_make_offer) info: Sending join-1 offer to rhel93 >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (do_dc_join_offer_all) info: Waiting on join-1 requests from 3 outstanding nodes >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (stonith__watchdog_fencing_enabled_for_node_api) notice: Cluster does not have watchdog fencing device >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.4.0 2 >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.0 (null) >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=5 >Aug 01 10:59:43.165 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="cluster1"/> >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (update_dc) info: Set DC to rhel92 (3.16.2) >Aug 01 10:59:43.165 rhel92 pacemaker-controld [1478938] (pcmk__update_peer_expected) info: update_dc: Node rhel92[2] - expected state is now member (was (null)) >Aug 01 10:59:43.166 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=rhel92/crmd/17, version=0.5.0) >Aug 01 10:59:43.166 rhel92 pacemaker-controld [1478938] (stonith__watchdog_fencing_enabled_for_node_api) notice: Cluster does not have watchdog fencing device >Aug 01 10:59:43.166 rhel92 pacemaker-controld [1478938] (stonith__watchdog_fencing_enabled_for_node_api) notice: Cluster does not have watchdog fencing device >Aug 01 10:59:43.167 rhel92 pacemaker-controld [1478938] (pcmk__update_peer_expected) info: do_dc_join_filter_offer: Node rhel93[3] - expected state is now member (was (null)) >Aug 01 10:59:43.167 rhel92 pacemaker-controld [1478938] (pcmk__update_peer_expected) info: do_dc_join_filter_offer: Node rhel91[1] - expected state is now member (was (null)) >Aug 01 10:59:43.167 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state >Aug 01 10:59:43.167 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/23) >Aug 01 10:59:43.167 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/24) >Aug 01 10:59:43.167 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/25) >Aug 01 10:59:43.167 rhel92 pacemaker-based [1478933] (cib_process_replace) info: Digest matched on replace from rhel92: 168cc25e1ac95f5ce314e1cdd0adbf01 >Aug 01 10:59:43.167 rhel92 pacemaker-based [1478933] (cib_process_replace) info: Replaced 0.5.0 with 0.5.0 from rhel92 >Aug 01 10:59:43.167 rhel92 pacemaker-controld [1478938] (connect_and_send_attrd_request) info: Connecting to cluster... 5 retries remaining >Aug 01 10:59:43.168 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=rhel92/crmd/22, version=0.5.0) >Aug 01 10:59:43.168 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=rhel92/crmd/23, version=0.5.0) >Aug 01 10:59:43.168 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=rhel92/crmd/24, version=0.5.0) >Aug 01 10:59:43.168 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=rhel92/crmd/25, version=0.5.0) >Aug 01 10:59:43.169 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #feature-set[rhel92]: (unset) -> 3.16.2 | from rhel92 with no write delay >Aug 01 10:59:43.169 rhel92 pacemaker-controld [1478938] (controld_delete_node_state) info: Deleting resource history for node rhel92 (via CIB call 26) | xpath=//node_state[@uname='rhel92']/lrm >Aug 01 10:59:43.169 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='rhel92']/lrm to all (origin=local/crmd/26) >Aug 01 10:59:43.169 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/27) >Aug 01 10:59:43.169 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #feature-set[rhel93]: (unset) -> 3.16.2 | from rhel93 with no write delay >Aug 01 10:59:43.169 rhel92 pacemaker-controld [1478938] (controld_delete_node_state) info: Deleting resource history for node rhel93 (via CIB call 28) | xpath=//node_state[@uname='rhel93']/lrm >Aug 01 10:59:43.169 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='rhel93']/lrm to all (origin=local/crmd/28) >Aug 01 10:59:43.169 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/29) >Aug 01 10:59:43.170 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting #feature-set[rhel91]: (unset) -> 3.16.2 | from rhel91 with no write delay >Aug 01 10:59:43.170 rhel92 pacemaker-controld [1478938] (controld_delete_node_state) info: Deleting resource history for node rhel91 (via CIB call 30) | xpath=//node_state[@uname='rhel91']/lrm >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rhel92']/lrm: OK (rc=0, origin=rhel92/crmd/26, version=0.5.0) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.0 2 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.1 (null) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=1 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status: <node_state id="2" uname="rhel92" in_ccm="true" crmd="online" crm-debug-origin="controld_query_executor_state"/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm id="2"> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_resources/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </node_state> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/27, version=0.5.1) >Aug 01 10:59:43.170 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rhel93']/lrm: OK (rc=0, origin=rhel92/crmd/28, version=0.5.1) >Aug 01 10:59:43.170 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition -1 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.1 2 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.2 (null) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=2 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status: <node_state id="3" uname="rhel93" in_ccm="true" crmd="online" crm-debug-origin="controld_query_executor_state"/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm id="3"> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_resources/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </node_state> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/29, version=0.5.2) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.2 2 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.3 (null) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=3 >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <instance_attributes id="status-2"> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="status-2-.feature-set" name="#feature-set" value="3.16.2"/> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </instance_attributes> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </transient_attributes> >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/attrd/4, version=0.5.3) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='rhel91']/lrm to all (origin=local/crmd/30) >Aug 01 10:59:43.170 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition -1 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.5.3 source=abort_unless_down:299 path=/cib/status/node_state[@id='2'] complete=true >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/31) >Aug 01 10:59:43.170 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/35) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/36) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/37) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rhel91']/lrm: OK (rc=0, origin=rhel92/crmd/30, version=0.5.3) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.3 2 >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.4 (null) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=4 >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status: <node_state id="1" uname="rhel91" in_ccm="true" crmd="online" crm-debug-origin="controld_query_executor_state"/> >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm id="1"> >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_resources/> >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm> >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </node_state> >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/31, version=0.5.4) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=rhel92/crmd/35, version=0.5.4) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.4 2 >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.5 (null) >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=5 >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition, @join=member, @expected=member >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']: @crm-debug-origin=do_state_transition, @join=member, @expected=member >Aug 01 10:59:43.171 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition, @join=member, @expected=member >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/36, version=0.5.5) >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.5 2 >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.6 (null) >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=6, @have-quorum=1, @dc-uuid=2 >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=rhel92/crmd/37, version=0.5.6) >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.6 2 >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.5.7 (null) >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=7 >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']: <transient_attributes id="3"/> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <instance_attributes id="status-3"> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="status-3-.feature-set" name="#feature-set" value="3.16.2"/> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </instance_attributes> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </transient_attributes> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <instance_attributes id="status-1"> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="status-1-.feature-set" name="#feature-set" value="3.16.2"/> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </instance_attributes> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </transient_attributes> >Aug 01 10:59:43.172 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/attrd/5, version=0.5.7) >Aug 01 10:59:43.181 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw >Aug 01 10:59:43.190 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.3.0 of the CIB to disk (digest: 49e532c4f89fe3b1e2ee59b64e757f7e) >Aug 01 10:59:43.197 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.e14bWT (digest: /var/lib/pacemaker/cib/cib.bmA6s1) >Aug 01 10:59:43.213 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw >Aug 01 10:59:43.222 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.5.0 of the CIB to disk (digest: 0933e218a09d4aea1d82c8e59ac79537) >Aug 01 10:59:43.231 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Vo3TZq (digest: /var/lib/pacemaker/cib/cib.NBOgBA) >Aug 01 10:59:44.177 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition -1 aborted by transient_attributes.3 'create': Transient attribute change | cib=0.5.7 source=abort_unless_down:299 path=/cib/status/node_state[@id='3'] complete=true >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (unpack_resources) error: Resource start-up disabled since no STONITH resources have been defined >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (unpack_resources) error: Either configure some or disable STONITH with the stonith-enabled option >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (unpack_resources) error: NOTE: Clusters with shared data need STONITH to ensure data integrity >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status_fencing) info: rhel92 is active >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status_fencing) info: rhel93 is active >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status_fencing) info: rhel91 is active >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 10:59:44.182 rhel92 pacemaker-schedulerd[1478937] (schedule_fencing_and_shutdowns) notice: No fencing will be done until there are resources to manage >Aug 01 10:59:44.183 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 0, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2 >Aug 01 10:59:44.183 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Configuration errors found during scheduler processing, please run "crm_verify -L" to identify issues >Aug 01 10:59:44.191 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 10:59:44.191 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 0 (ref=pe_calc-pacemaker-controld-1690912784-15) derived from /var/lib/pacemaker/pengine/pe-input-2.bz2 >Aug 01 10:59:44.191 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 0 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete >Aug 01 10:59:44.191 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 10:59:44.191 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 10:59:48.174 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: 740c6722418b824887dd3d3b0cece4be for 0.5.7 (0x55c8f0fed6b0 0) >Aug 01 10:59:48.367 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.5.7 2 >Aug 01 10:59:48.367 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.6.0 (null) >Aug 01 10:59:48.367 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=6, @num_updates=0 >Aug 01 10:59:48.367 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/> >Aug 01 10:59:48.367 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=rhel91/cibadmin/2, version=0.6.0) >Aug 01 10:59:48.367 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 0 aborted by cib-bootstrap-options-stonith-enabled doing create stonith-enabled=false: Configuration change | cib=0.6.0 source=te_update_diff_v2:465 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options'] complete=true >Aug 01 10:59:48.367 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 10:59:48.382 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-4.raw >Aug 01 10:59:48.397 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.6.0 of the CIB to disk (digest: 1f5708c803f75d9fa69d1cbba8fd9f52) >Aug 01 10:59:48.411 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.pozuCE (digest: /var/lib/pacemaker/cib/cib.RohE26) >Aug 01 10:59:49.372 rhel92 pacemaker-controld [1478938] (stonith__watchdog_fencing_enabled_for_node_api) notice: Cluster does not have watchdog fencing device >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (unpack_config) warning: Blind faith: not fencing unseen nodes >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (schedule_fencing_and_shutdowns) notice: No fencing will be done until there are resources to manage >Aug 01 10:59:49.375 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-3.bz2 >Aug 01 10:59:49.384 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 10:59:49.384 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 1 (ref=pe_calc-pacemaker-controld-1690912789-16) derived from /var/lib/pacemaker/pengine/pe-input-3.bz2 >Aug 01 10:59:49.384 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 1 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-3.bz2): Complete >Aug 01 10:59:49.384 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 10:59:49.384 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 10:59:52.151 rhel92 pacemaker-controld [1478938] (pcmk__procfs_pid_of) info: Found pacemaker-based active as process 1478933 >Aug 01 10:59:52.151 rhel92 pacemaker-controld [1478938] (throttle_send_command) info: New throttle mode: negligible load (was undetermined) >Aug 01 10:59:53.369 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: b3bb189ab68034c5fc0cd792d2f64a35 for 0.6.0 (0x55c8f0ffb7e0 0) >Aug 01 11:03:02.015 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.6.0 2 >Aug 01 11:03:02.015 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.7.0 dff1077c6fc2630bed8414810d829c08 >Aug 01 11:03:02.015 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=7 >Aug 01 11:03:02.015 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-start-failure-is-fat" name="start-failure-is-fatal" value="true"/> >Aug 01 11:03:02.015 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=rhel91/cibadmin/2, version=0.7.0) >Aug 01 11:03:02.015 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 1 aborted by cib-bootstrap-options-start-failure-is-fat doing create start-failure-is-fatal=true: Configuration change | cib=0.7.0 source=te_update_diff_v2:465 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options'] complete=true >Aug 01 11:03:02.015 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 11:03:02.032 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-5.raw >Aug 01 11:03:02.047 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.7.0 of the CIB to disk (digest: d837455833b73c6cba601d02343263d5) >Aug 01 11:03:02.062 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0eIvaN (digest: /var/lib/pacemaker/cib/cib.pUNJOG) >Aug 01 11:03:03.021 rhel92 pacemaker-controld [1478938] (stonith__watchdog_fencing_enabled_for_node_api) notice: Cluster does not have watchdog fencing device >Aug 01 11:03:03.024 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:03:03.024 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:03:03.024 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:03:03.024 rhel92 pacemaker-schedulerd[1478937] (schedule_fencing_and_shutdowns) notice: No fencing will be done until there are resources to manage >Aug 01 11:03:03.025 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-4.bz2 >Aug 01 11:03:03.031 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:03:03.031 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 2 (ref=pe_calc-pacemaker-controld-1690912983-18) derived from /var/lib/pacemaker/pengine/pe-input-4.bz2 >Aug 01 11:03:03.031 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 2 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete >Aug 01 11:03:03.031 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 11:03:03.031 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:03:07.017 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: dff1077c6fc2630bed8414810d829c08 for 0.7.0 (0x55c8f0f65100 0) >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.7.0 2 >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.0 (null) >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=8 >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/resources: <clone id="ag_cluster-clone"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <primitive id="ag_cluster" class="ocf" type="ag" provider="mssql"> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <instance_attributes id="ag_cluster-instance_attributes"> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="ag_cluster-instance_attributes-ag_name" name="ag_name" value="ag1"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </instance_attributes> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <meta_attributes id="ag_cluster-meta_attributes"> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="ag_cluster-meta_attributes-failure-timeout" name="failure-timeout" value="60s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </meta_attributes> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <operations> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="demote" interval="0s" timeout="10" id="ag_cluster-demote-interval-0s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="monitor" interval="10" timeout="60" id="ag_cluster-monitor-interval-10"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="monitor" interval="11" timeout="60" role="Promoted" id="ag_cluster-monitor-interval-11"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="monitor" interval="12" timeout="60" role="Unpromoted" id="ag_cluster-monitor-interval-12"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="notify" interval="0s" timeout="60" id="ag_cluster-notify-interval-0s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="promote" interval="0s" timeout="60" id="ag_cluster-promote-interval-0s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="start" interval="0s" timeout="60" id="ag_cluster-start-interval-0s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="stop" interval="0s" timeout="10" id="ag_cluster-stop-interval-0s"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </operations> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </primitive> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <meta_attributes id="ag_cluster-clone-meta_attributes"> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="ag_cluster-clone-meta_attributes-notify" name="notify" value="true"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="ag_cluster-clone-meta_attributes-promotable" name="promotable" value="true"/> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </meta_attributes> >Aug 01 11:03:42.096 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </clone> >Aug 01 11:03:42.097 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=rhel91/cibadmin/2, version=0.8.0) >Aug 01 11:03:42.097 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 2 aborted by clone.ag_cluster-clone 'create': Configuration change | cib=0.8.0 source=te_update_diff_v2:465 path=/cib/configuration/resources complete=true >Aug 01 11:03:42.097 rhel92 pacemaker-fenced [1478934] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources >Aug 01 11:03:42.097 rhel92 pacemaker-fenced [1478934] (cib_devices_update) info: Updating devices to version 0.8.0 >Aug 01 11:03:42.097 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 11:03:42.097 rhel92 pacemaker-fenced [1478934] (unpack_config) warning: Blind faith: not fencing unseen nodes >Aug 01 11:03:42.101 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:03:42.101 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:03:42.101 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:03:42.101 rhel92 pacemaker-schedulerd[1478937] (pcmk__set_instance_roles) info: ag_cluster-clone: Promoted 0 instances of a possible 1 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:0 on rhel91 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:0 on rhel91 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:1 on rhel92 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:1 on rhel92 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:2 on rhel93 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:2 on rhel93 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:0 on rhel91 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:0 on rhel91 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:1 on rhel92 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:1 on rhel92 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:2 on rhel93 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:2 on rhel93 >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (log_list_item) notice: Actions: Start ag_cluster:0 ( rhel91 ) >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (log_list_item) notice: Actions: Start ag_cluster:1 ( rhel92 ) >Aug 01 11:03:42.102 rhel92 pacemaker-schedulerd[1478937] (log_list_item) notice: Actions: Start ag_cluster:2 ( rhel93 ) >Aug 01 11:03:42.103 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-5.bz2 >Aug 01 11:03:42.109 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-6.raw >Aug 01 11:03:42.119 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:03:42.120 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 3 (ref=pe_calc-pacemaker-controld-1690913022-19) derived from /var/lib/pacemaker/pengine/pe-input-5.bz2 >Aug 01 11:03:42.120 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster:0_monitor_0 on rhel91 | action 1 >Aug 01 11:03:42.120 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster:1_monitor_0 locally on rhel92 | action 2 >Aug 01 11:03:42.120 rhel92 pacemaker-execd [1478935] (process_lrmd_get_rsc_info) info: Agent information for 'ag_cluster' not in cache >Aug 01 11:03:42.120 rhel92 pacemaker-execd [1478935] (process_lrmd_get_rsc_info) info: Agent information for 'ag_cluster:1' not in cache >Aug 01 11:03:42.120 rhel92 pacemaker-execd [1478935] (process_lrmd_rsc_register) info: Cached agent information for 'ag_cluster' >Aug 01 11:03:42.121 rhel92 pacemaker-controld [1478938] (do_lrm_invoke) info: Retrieving metadata for ag_cluster (ocf:mssql:ag) asynchronously >Aug 01 11:03:42.121 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster:2_monitor_0 on rhel93 | action 3 >Aug 01 11:03:42.126 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.8.0 of the CIB to disk (digest: 07233e891f5f4a0d830ecf5331438c6f) >Aug 01 11:03:42.127 rhel92 pacemaker-controld [1478938] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for ag_cluster on rhel92 | transition_key=2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 op_key=ag_cluster_monitor_0 >Aug 01 11:03:42.128 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/49) >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.0 2 >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.1 (null) >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=1 >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="ag_cluster" type="ag" class="ocf" provider="mssql"/> >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="ag_cluster_last_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel92" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913022" exec-time="0" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:03:42.129 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/49, version=0.8.1) >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.1 2 >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.2 (null) >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=2 >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="ag_cluster" type="ag" class="ocf" provider="mssql"/> >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="ag_cluster_last_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel91" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913022" exec-time="0" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:03:42.130 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/14, version=0.8.2) >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.2 2 >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.3 (null) >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=3 >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']: @crm-debug-origin=do_update_resource >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources: <lrm_resource id="ag_cluster" type="ag" class="ocf" provider="mssql"/> >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="ag_cluster_last_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel93" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913022" exec-time="0" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:03:42.131 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/14, version=0.8.3) >Aug 01 11:03:42.137 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Ur2wcw (digest: /var/lib/pacemaker/cib/cib.BEyIlq) >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: mssql_validate >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: Resource agent invoked with: monitor >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: mssql_monitor >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:42 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]; monitor-interval-timeout [60] >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:42 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:03:42 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:42 Connected to the instance at localhost:1433 >Aug 01 11:03:47.132 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: e24fc365f122f4fd6e283a3dd3c79466 for 0.8.3 (0x55c8f1003d30 0) >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:47 Monitor Caller is: monitor. >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:47 Replica is SECONDARY (2) >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:47 Instance name is rhel92. >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: 2023/08/01 11:03:47 Current master is . >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:03:47.253 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel92]: (unset) -> | from rhel92 with no write delay >Aug 01 11:03:47.253 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel93]: (unset) -> | from rhel93 with no write delay >Aug 01 11:03:47.257 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: (unset) -> 20230801180449 | from rhel91 with no write delay >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: lease_expiry after monitor update: >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >Aug 01 11:03:47.276 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting master-ag_cluster[rhel92]: (unset) -> 10 | from rhel92 with no write delay >Aug 01 11:03:47.277 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting master-ag_cluster[rhel93]: (unset) -> 10 | from rhel93 with no write delay >Aug 01 11:03:47.277 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.3 2 >Aug 01 11:03:47.277 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.4 (null) >Aug 01 11:03:47.277 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=4 >Aug 01 11:03:47.277 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-master-ag_cluster" name="master-ag_cluster" value="10"/> >Aug 01 11:03:47.277 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/attrd/6, version=0.8.4) >Aug 01 11:03:47.278 rhel92 pacemaker-controld [1478938] (abort_transition_graph) notice: Transition 3 aborted by status-2-master-ag_cluster doing create master-ag_cluster=10: Transient attribute change | cib=0.8.4 source=abort_unless_down:299 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false >Aug 01 11:03:47.278 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.4 2 >Aug 01 11:03:47.278 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.5 (null) >Aug 01 11:03:47.278 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=5 >Aug 01 11:03:47.278 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/transient_attributes[@id='3']/instance_attributes[@id='status-3']: <nvpair id="status-3-master-ag_cluster" name="master-ag_cluster" value="10"/> >Aug 01 11:03:47.278 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/attrd/7, version=0.8.5) >Aug 01 11:03:47.278 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 3 aborted by status-3-master-ag_cluster doing create master-ag_cluster=10: Transient attribute change | cib=0.8.5 source=abort_unless_down:299 path=/cib/status/node_state[@id='3']/transient_attributes[@id='3']/instance_attributes[@id='status-3'] complete=false >Aug 01 11:03:47 ag(ag_cluster)[1479060]: INFO: ag_cluster monitor : 0 >Aug 01 11:03:47.280 rhel92 pacemaker-execd [1478935] (log_op_output) info: ag_cluster_monitor_0[1479060] error output [ resource ag_cluster is NOT running ] >Aug 01 11:03:47.280 rhel92 pacemaker-controld [1478938] (log_executor_event) notice: Result of probe operation for ag_cluster on rhel92: ok | CIB update 50, graph action confirmed; call=6 key=ag_cluster_monitor_0 rc=0 >Aug 01 11:03:47.280 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/50) >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.5 2 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.6 (null) >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=6 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_last_0']: @transition-magic=0:0;2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=6, @rc-code=0, @op-status=0, @exec-time=5152 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_last_failure_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="0:0;2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel92" call-id="6" rc-code="0" op-status="0" interval="0" last-rc-change="1690913022" exec-time="5152" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/50, version=0.8.6) >Aug 01 11:03:47.281 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 3 aborted by operation ag_cluster_monitor_0 'modify' on rhel92: Event failed | magic=0:0;2:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 cib=0.8.6 source=process_graph_event:490 complete=false >Aug 01 11:03:47.281 rhel92 pacemaker-controld [1478938] (process_graph_event) notice: Transition 3 action 2 (ag_cluster_monitor_0 on rhel92): expected 'not running' but got 'ok' | target-rc=7 rc=0 call-id=6 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.6 2 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.7 (null) >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=7 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_last_0']: @transition-magic=0:0;3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=6, @rc-code=0, @op-status=0, @exec-time=5149 >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_last_failure_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="0:0;3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel93" call-id="6" rc-code="0" op-status="0" interval="0" last-rc-change="1690913022" exec-time="5149" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:47.281 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/15, version=0.8.7) >Aug 01 11:03:47.281 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 3 aborted by operation ag_cluster_monitor_0 'modify' on rhel93: Event failed | magic=0:0;3:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 cib=0.8.7 source=process_graph_event:490 complete=false >Aug 01 11:03:47.281 rhel92 pacemaker-controld [1478938] (process_graph_event) notice: Transition 3 action 3 (ag_cluster_monitor_0 on rhel93): expected 'not running' but got 'ok' | target-rc=7 rc=0 call-id=6 >Aug 01 11:03:47.282 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting master-ag_cluster[rhel91]: (unset) -> 20 | from rhel91 with no write delay >Aug 01 11:03:47.283 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.7 2 >Aug 01 11:03:47.283 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.8 (null) >Aug 01 11:03:47.283 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=8 >Aug 01 11:03:47.283 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-master-ag_cluster" name="master-ag_cluster" value="20"/> >Aug 01 11:03:47.283 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/attrd/8, version=0.8.8) >Aug 01 11:03:47.283 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 3 aborted by status-1-master-ag_cluster doing create master-ag_cluster=20: Transient attribute change | cib=0.8.8 source=abort_unless_down:299 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.8 2 >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.9 (null) >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=9 >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_last_0']: @transition-magic=0:8;1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=6, @rc-code=8, @op-status=0, @exec-time=5155 >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_last_failure_0" operation_key="ag_cluster_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="0:8;1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel91" call-id="6" rc-code="8" op-status="0" interval="0" last-rc-change="1690913022" exec-time="5155" queue-time="0" op-digest="9672526152b2afa3a0cf05537be3c379"/> >Aug 01 11:03:47.286 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/15, version=0.8.9) >Aug 01 11:03:47.286 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 3 aborted by operation ag_cluster_monitor_0 'modify' on rhel91: Event failed | magic=0:8;1:3:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 cib=0.8.9 source=process_graph_event:490 complete=false >Aug 01 11:03:47.286 rhel92 pacemaker-controld [1478938] (process_graph_event) notice: Transition 3 action 1 (ag_cluster_monitor_0 on rhel91): expected 'not running' but got 'promoted' | target-rc=7 rc=8 call-id=6 >Aug 01 11:03:47.286 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 3 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=16, Source=/var/lib/pacemaker/pengine/pe-input-5.bz2): Complete >Aug 01 11:03:47.286 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.290 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (set_instance_role) info: Choosing ag_cluster:2 (Promoted) on rhel91 for promotion >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (pcmk__set_instance_roles) info: ag_cluster-clone: Promoted 1 instances of a possible 1 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:0 on rhel92 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:0 on rhel92 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:1 on rhel93 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:1 on rhel93 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 11s-interval monitor for ag_cluster:2 on rhel91 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:0 on rhel92 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:0 on rhel92 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for ag_cluster:1 on rhel93 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 12s-interval monitor for ag_cluster:1 on rhel93 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 11s-interval monitor for ag_cluster:2 on rhel91 >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:0 (Unpromoted rhel92) >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:1 (Unpromoted rhel93) >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:2 (Promoted rhel91) >Aug 01 11:03:47.291 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-6.bz2 >Aug 01 11:03:47.301 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:03:47.301 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 4 (ref=pe_calc-pacemaker-controld-1690913027-23) derived from /var/lib/pacemaker/pengine/pe-input-6.bz2 >Aug 01 11:03:47.301 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster_monitor_12000 locally on rhel92 | action 4 >Aug 01 11:03:47.301 rhel92 pacemaker-controld [1478938] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ag_cluster on rhel92 | transition_key=4:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 op_key=ag_cluster_monitor_12000 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/52) >Aug 01 11:03:47.302 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster_monitor_10000 locally on rhel92 | action 3 >Aug 01 11:03:47.302 rhel92 pacemaker-controld [1478938] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ag_cluster on rhel92 | transition_key=3:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 op_key=ag_cluster_monitor_10000 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/53) >Aug 01 11:03:47.302 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster_monitor_12000 on rhel93 | action 8 >Aug 01 11:03:47.302 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster_monitor_10000 on rhel93 | action 7 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.9 2 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.10 (null) >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=10 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_monitor_12000" operation_key="ag_cluster_monitor_12000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="4:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;4:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel92" call-id="-1" rc-code="193" op-status="-1" interval="12000" last-rc-change="1690913027" exec-time="0" queue-time="0" op-digest="a5ccb527549b0da2ccce481bc9858999"/> >Aug 01 11:03:47.302 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation ag_cluster_monitor_11000 on rhel91 | action 13 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/52, version=0.8.10) >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.10 2 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.11 (null) >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=11 >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_monitor_10000" operation_key="ag_cluster_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="3:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;3:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel92" call-id="-1" rc-code="193" op-status="-1" interval="10000" last-rc-change="1690913027" exec-time="0" queue-time="0" op-digest="a5ccb527549b0da2ccce481bc9858999"/> >Aug 01 11:03:47.302 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/53, version=0.8.11) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.11 2 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.12 (null) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=12 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_monitor_12000" operation_key="ag_cluster_monitor_12000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="8:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;8:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel93" call-id="-1" rc-code="193" op-status="-1" interval="12000" last-rc-change="1690913027" exec-time="0" queue-time="0" op-digest="a5ccb527549b0da2ccce481bc9858999"/> >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/16, version=0.8.12) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.12 2 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.13 (null) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=13 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_monitor_11000" operation_key="ag_cluster_monitor_11000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="13:4:8:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;13:4:8:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel91" call-id="-1" rc-code="193" op-status="-1" interval="11000" last-rc-change="1690913027" exec-time="0" queue-time="0" op-digest="a5ccb527549b0da2ccce481bc9858999"/> >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/16, version=0.8.13) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.13 2 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.14 (null) >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=14 >Aug 01 11:03:47.304 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']: <lrm_rsc_op id="ag_cluster_monitor_10000" operation_key="ag_cluster_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="7:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;7:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel93" call-id="-1" rc-code="193" op-status="-1" interval="10000" last-rc-change="1690913027" exec-time="0" queue-time="0" op-digest="a5ccb527549b0da2ccce481bc9858999"/> >Aug 01 11:03:47.305 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/17, version=0.8.14) >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: mssql_validate >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: Resource agent invoked with: monitor >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: mssql_monitor >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:03:47 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:47 Connected to the instance at localhost:1433 >Aug 01 11:03:52.306 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: 74b4f3ce718b94ce96090af12715c31b for 0.8.14 (0x55c8f1003d30 0) >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:52 Monitor Caller is: monitor. >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:52 Replica is SECONDARY (2) >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:52 Instance name is rhel92. >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: 2023/08/01 11:03:52 Current master is . >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:03:52.414 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180449 -> 20230801180505 | from rhel91 with no write delay >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: lease_expiry after monitor update: >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180505" >Aug 01 11:03:52 ag(ag_cluster)[1479176]: INFO: ag_cluster monitor : 0 >Aug 01 11:03:52.440 rhel92 pacemaker-controld [1478938] (log_executor_event) notice: Result of monitor operation for ag_cluster on rhel92: ok | CIB update 54, graph action unconfirmed; call=7 key=ag_cluster_monitor_12000 rc=0 >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/54) >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.14 2 >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.15 (null) >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=15 >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_monitor_12000']: @transition-magic=0:0;4:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=7, @rc-code=0, @op-status=0, @exec-time=5137 >Aug 01 11:03:52.440 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/54, version=0.8.15) >Aug 01 11:03:52.440 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 4 action 4 (ag_cluster_monitor_12000 on rhel92) confirmed: ok | rc=0 call-id=7 >Aug 01 11:03:52.441 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.15 2 >Aug 01 11:03:52.441 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.16 (null) >Aug 01 11:03:52.441 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=16 >Aug 01 11:03:52.441 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_monitor_12000']: @transition-magic=0:0;8:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=7, @rc-code=0, @op-status=0, @exec-time=5136 >Aug 01 11:03:52.441 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/18, version=0.8.16) >Aug 01 11:03:52.441 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 4 action 8 (ag_cluster_monitor_12000 on rhel93) confirmed: ok | rc=0 call-id=7 >Aug 01 11:03:52.445 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.16 2 >Aug 01 11:03:52.445 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.17 (null) >Aug 01 11:03:52.445 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=17 >Aug 01 11:03:52.445 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_monitor_11000']: @transition-magic=0:8;13:4:8:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=7, @rc-code=8, @op-status=0, @last-rc-change=1690913032, @exec-time=5140 >Aug 01 11:03:52.445 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/17, version=0.8.17) >Aug 01 11:03:52.445 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 4 action 13 (ag_cluster_monitor_11000 on rhel91) confirmed: promoted | rc=8 call-id=7 >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: mssql_validate >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: Resource agent invoked with: monitor >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: mssql_monitor >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:03:52 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:52 Connected to the instance at localhost:1433 >Aug 01 11:03:57.447 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: 4c1f78ca1222aa8abd6c94f4e21208ad for 0.8.17 (0x55c8f1003d30 0) >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:57 Monitor Caller is: monitor. >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:57 Replica is SECONDARY (2) >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:57 Instance name is rhel92. >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: 2023/08/01 11:03:57 Current master is . >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: lease_expiry after monitor update: >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180505" >Aug 01 11:03:57 ag(ag_cluster)[1479292]: INFO: ag_cluster monitor : 0 >Aug 01 11:03:57.572 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.17 2 >Aug 01 11:03:57.572 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.18 (null) >Aug 01 11:03:57.572 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=18 >Aug 01 11:03:57.572 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_monitor_10000']: @transition-magic=0:0;7:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=8, @rc-code=0, @op-status=0, @last-rc-change=1690913032, @exec-time=5131, @queue-time=5135 >Aug 01 11:03:57.572 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/19, version=0.8.18) >Aug 01 11:03:57.573 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 4 action 7 (ag_cluster_monitor_10000 on rhel93) confirmed: ok | rc=0 call-id=8 >Aug 01 11:03:57.573 rhel92 pacemaker-controld [1478938] (log_executor_event) notice: Result of monitor operation for ag_cluster on rhel92: ok | CIB update 55, graph action unconfirmed; call=8 key=ag_cluster_monitor_10000 rc=0 >Aug 01 11:03:57.573 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/55) >Aug 01 11:03:57.573 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.18 2 >Aug 01 11:03:57.573 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.8.19 (null) >Aug 01 11:03:57.573 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=19 >Aug 01 11:03:57.573 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ag_cluster']/lrm_rsc_op[@id='ag_cluster_monitor_10000']: @transition-magic=0:0;3:4:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=8, @rc-code=0, @op-status=0, @last-rc-change=1690913032, @exec-time=5133, @queue-time=5137 >Aug 01 11:03:57.574 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/55, version=0.8.19) >Aug 01 11:03:57.574 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 4 action 3 (ag_cluster_monitor_10000 on rhel92) confirmed: ok | rc=0 call-id=8 >Aug 01 11:03:57.574 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 4 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-6.bz2): Complete >Aug 01 11:03:57.574 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 11:03:57.574 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:04:02.575 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: ddb7101e1285be69a9c9d05d45e85f76 for 0.8.19 (0x55c8f1003d30 0) >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: mssql_validate >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: mssql_monitor >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:04 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:04 Connected to the instance at localhost:1433 >Aug 01 11:04:08.549 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180505 -> 20230801180521 | from rhel91 with no write delay >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:09 Monitor Caller is: monitor. >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:09 Replica is SECONDARY (2) >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:09 Instance name is rhel92. >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: 2023/08/01 11:04:09 Current master is . >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: lease_expiry after monitor update: >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180521" >Aug 01 11:04:09 ag(ag_cluster)[1479412]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: mssql_validate >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: mssql_monitor >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:09 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:09 Connected to the instance at localhost:1433 >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:14 Monitor Caller is: monitor. >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:14 Replica is SECONDARY (2) >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:14 Instance name is rhel92. >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: 2023/08/01 11:04:14 Current master is . >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: lease_expiry after monitor update: >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180521" >Aug 01 11:04:14 ag(ag_cluster)[1479529]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: mssql_validate >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: mssql_monitor >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:21 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:21 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:21 Connected to the instance at localhost:1433 >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.8.19 2 >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.0 0d7ff4fc7e37c18d7de70e1dc3b1dfcf >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=9, @num_updates=0 >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/resources: <primitive id="virtualip" class="ocf" type="IPaddr2" provider="heartbeat"/> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <instance_attributes id="virtualip-instance_attributes"> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <nvpair id="virtualip-instance_attributes-ip" name="ip" value="10.135.65.30"/> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </instance_attributes> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <operations> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="monitor" interval="10s" timeout="20s" id="virtualip-monitor-interval-10s"/> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="start" interval="0s" timeout="20s" id="virtualip-start-interval-0s"/> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <op name="stop" interval="0s" timeout="20s" id="virtualip-stop-interval-0s"/> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </operations> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </primitive> >Aug 01 11:04:22.682 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=rhel91/cibadmin/2, version=0.9.0) >Aug 01 11:04:22.682 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 4 aborted by primitive.virtualip 'create': Configuration change | cib=0.9.0 source=te_update_diff_v2:465 path=/cib/configuration/resources complete=true >Aug 01 11:04:22.682 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 11:04:22.683 rhel92 pacemaker-fenced [1478934] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources >Aug 01 11:04:22.683 rhel92 pacemaker-fenced [1478934] (cib_devices_update) info: Updating devices to version 0.9.0 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.687 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (set_instance_role) info: Choosing ag_cluster:2 (Promoted) on rhel91 for promotion >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (pcmk__set_instance_roles) info: ag_cluster-clone: Promoted 1 instances of a possible 1 >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (recurring_op_for_active) info: Start 10s-interval monitor for virtualip on rhel91 >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:0 (Unpromoted rhel92) >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:1 (Unpromoted rhel93) >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:2 (Promoted rhel91) >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (log_list_item) notice: Actions: Start virtualip ( rhel91 ) >Aug 01 11:04:22.688 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-7.bz2 >Aug 01 11:04:22.698 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-7.raw >Aug 01 11:04:22.710 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:04:22.710 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 5 (ref=pe_calc-pacemaker-controld-1690913062-29) derived from /var/lib/pacemaker/pengine/pe-input-7.bz2 >Aug 01 11:04:22.710 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation virtualip_monitor_0 on rhel93 | action 8 >Aug 01 11:04:22.711 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation virtualip_monitor_0 locally on rhel92 | action 7 >Aug 01 11:04:22.711 rhel92 pacemaker-execd [1478935] (process_lrmd_get_rsc_info) info: Agent information for 'virtualip' not in cache >Aug 01 11:04:22.711 rhel92 pacemaker-execd [1478935] (process_lrmd_rsc_register) info: Cached agent information for 'virtualip' >Aug 01 11:04:22.711 rhel92 pacemaker-controld [1478938] (do_lrm_invoke) info: Retrieving metadata for virtualip (ocf:heartbeat:IPaddr2) asynchronously >Aug 01 11:04:22.711 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation virtualip_monitor_0 on rhel91 | action 6 >Aug 01 11:04:22.717 rhel92 pacemaker-controld [1478938] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for virtualip on rhel92 | transition_key=7:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4 op_key=virtualip_monitor_0 >Aug 01 11:04:22.717 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/57) >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.0 2 >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.1 (null) >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=1 >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="virtualip" type="IPaddr2" class="ocf" provider="heartbeat"/> >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="virtualip_last_0" operation_key="virtualip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="7:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;7:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel92" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913062" exec-time="0" queue-time="0" op-digest="1875668557da4ce1d39bcf81dfed63ca"/> >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:04:22.718 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/57, version=0.9.1) >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.1 2 >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.2 (null) >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=2 >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources: <lrm_resource id="virtualip" type="IPaddr2" class="ocf" provider="heartbeat"/> >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="virtualip_last_0" operation_key="virtualip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="8:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;8:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel93" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913062" exec-time="0" queue-time="0" op-digest="1875668557da4ce1d39bcf81dfed63ca"/> >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:04:22.720 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/20, version=0.9.2) >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.9.0 of the CIB to disk (digest: d8d0e1f69d8b4087b6cae23b079d7518) >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.2 2 >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.3 (null) >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=3 >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="virtualip" type="IPaddr2" class="ocf" provider="heartbeat"/> >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ <lrm_rsc_op id="virtualip_last_0" operation_key="virtualip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="6:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;6:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel91" call-id="-1" rc-code="193" op-status="-1" interval="0" last-rc-change="1690913062" exec-time="0" queue-time="0" op-digest="1875668557da4ce1d39bcf81dfed63ca"/> >Aug 01 11:04:22.721 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ </lrm_resource> >Aug 01 11:04:22.722 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/18, version=0.9.3) >Aug 01 11:04:22.736 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2MTP0I (digest: /var/lib/pacemaker/cib/cib.221sAt) >Aug 01 11:04:22.740 rhel92 pacemaker-controld [1478938] (log_executor_event) notice: Result of probe operation for virtualip on rhel92: not running | CIB update 58, graph action confirmed; call=12 key=virtualip_monitor_0 rc=7 >Aug 01 11:04:22.740 rhel92 pacemaker-based [1478933] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/58) >Aug 01 11:04:22.741 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.3 2 >Aug 01 11:04:22.741 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.4 (null) >Aug 01 11:04:22.741 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=4 >Aug 01 11:04:22.741 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_last_0']: @transition-magic=0:7;7:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=12, @rc-code=7, @op-status=0, @exec-time=22 >Aug 01 11:04:22.741 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel92/crmd/58, version=0.9.4) >Aug 01 11:04:22.741 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 5 action 7 (virtualip_monitor_0 on rhel92) confirmed: not running | rc=7 call-id=12 >Aug 01 11:04:22.743 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.4 2 >Aug 01 11:04:22.743 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.5 (null) >Aug 01 11:04:22.743 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=5 >Aug 01 11:04:22.743 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_last_0']: @transition-magic=0:7;6:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=11, @rc-code=7, @op-status=0, @exec-time=21 >Aug 01 11:04:22.743 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/19, version=0.9.5) >Aug 01 11:04:22.743 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 5 action 6 (virtualip_monitor_0 on rhel91) confirmed: not running | rc=7 call-id=11 >Aug 01 11:04:22.746 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.5 2 >Aug 01 11:04:22.746 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.6 (null) >Aug 01 11:04:22.746 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=6 >Aug 01 11:04:22.746 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_last_0']: @transition-magic=0:7;8:5:7:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=12, @rc-code=7, @op-status=0, @exec-time=25 >Aug 01 11:04:22.746 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel93/crmd/21, version=0.9.6) >Aug 01 11:04:22.746 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 5 action 8 (virtualip_monitor_0 on rhel93) confirmed: not running | rc=7 call-id=12 >Aug 01 11:04:22.746 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating start operation virtualip_start_0 on rhel91 | action 41 >Aug 01 11:04:22.754 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.6 2 >Aug 01 11:04:22.754 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.7 (null) >Aug 01 11:04:22.754 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=7 >Aug 01 11:04:22.754 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_last_0']: @operation_key=virtualip_start_0, @operation=start, @transition-key=41:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @transition-magic=-1:193;41:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 >Aug 01 11:04:22.754 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/20, version=0.9.7) >Aug 01 11:04:22.785 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.7 2 >Aug 01 11:04:22.785 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.8 (null) >Aug 01 11:04:22.785 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=8 >Aug 01 11:04:22.785 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_last_0']: @transition-magic=0:0;41:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=12, @rc-code=0, @op-status=0, @exec-time=30 >Aug 01 11:04:22.785 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/21, version=0.9.8) >Aug 01 11:04:22.785 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 5 action 41 (virtualip_start_0 on rhel91) confirmed: ok | rc=0 call-id=12 >Aug 01 11:04:22.785 rhel92 pacemaker-controld [1478938] (execute_rsc_action) notice: Initiating monitor operation virtualip_monitor_10000 on rhel91 | action 42 >Aug 01 11:04:22.787 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.8 2 >Aug 01 11:04:22.787 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.9 (null) >Aug 01 11:04:22.787 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=9 >Aug 01 11:04:22.787 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip']: <lrm_rsc_op id="virtualip_monitor_10000" operation_key="virtualip_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.16.2" transition-key="42:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" transition-magic="-1:193;42:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4" exit-reason="" on_node="rhel91" call-id="-1" rc-code="193" op-status="-1" interval="10000" last-rc-change="1690913062" exec-time="0" queue-time="0" op-digest="2c3f2afdd0017d809047a3ec083e680c"/> >Aug 01 11:04:22.787 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/22, version=0.9.9) >Aug 01 11:04:22.808 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.9 2 >Aug 01 11:04:22.808 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.9.10 (null) >Aug 01 11:04:22.808 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @num_updates=10 >Aug 01 11:04:22.808 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip']/lrm_rsc_op[@id='virtualip_monitor_10000']: @transition-magic=0:0;42:5:0:976395e4-0f30-4f7e-9b5f-1da0b320d7c4, @call-id=13, @rc-code=0, @op-status=0, @exec-time=20 >Aug 01 11:04:22.808 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=rhel91/crmd/23, version=0.9.10) >Aug 01 11:04:22.808 rhel92 pacemaker-controld [1478938] (process_graph_event) info: Transition 5 action 42 (virtualip_monitor_10000 on rhel91) confirmed: ok | rc=0 call-id=13 >Aug 01 11:04:22.808 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 5 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-7.bz2): Complete >Aug 01 11:04:22.808 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 11:04:22.808 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:04:24.679 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180521 -> 20230801180537 | from rhel91 with no write delay >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:26 Monitor Caller is: monitor. >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:26 Replica is SECONDARY (2) >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:26 Instance name is rhel92. >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: 2023/08/01 11:04:26 Current master is . >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: lease_expiry after monitor update: >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180537" >Aug 01 11:04:26 ag(ag_cluster)[1479648]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: mssql_validate >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: mssql_monitor >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:26 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:26 Connected to the instance at localhost:1433 >Aug 01 11:04:27.809 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: ef6beac77210229610a132c51fc0e0cc for 0.9.10 (0x55c8f101c490 0) >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:31 Monitor Caller is: monitor. >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:31 Replica is SECONDARY (2) >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:31 Instance name is rhel92. >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: 2023/08/01 11:04:31 Current master is . >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: lease_expiry after monitor update: >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180537" >Aug 01 11:04:31 ag(ag_cluster)[1479811]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: mssql_validate >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: mssql_monitor >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:38 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:38 Connected to the instance at localhost:1433 >Aug 01 11:04:40.806 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180537 -> 20230801180553 | from rhel91 with no write delay >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:43 Monitor Caller is: monitor. >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:43 Replica is SECONDARY (2) >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:43 Instance name is rhel92. >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: 2023/08/01 11:04:43 Current master is . >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: lease_expiry after monitor update: >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180553" >Aug 01 11:04:43 ag(ag_cluster)[1479928]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: mssql_validate >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: mssql_monitor >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:43 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:43 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:43 Connected to the instance at localhost:1433 >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:48 Monitor Caller is: monitor. >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:48 Replica is SECONDARY (2) >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:48 Instance name is rhel92. >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: 2023/08/01 11:04:48 Current master is . >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: lease_expiry after monitor update: >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180553" >Aug 01 11:04:48 ag(ag_cluster)[1480045]: INFO: ag_cluster monitor : 0 >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: mssql_validate >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: Resource agent invoked with: monitor >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: mssql_monitor >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:04:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:04:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:04:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:04:55 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:04:55 Connected to the instance at localhost:1433 >Aug 01 11:04:56.932 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180553 -> 20230801180609 | from rhel91 with no write delay >Aug 01 11:05:00.218 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.9.10 2 >Aug 01 11:05:00.218 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.10.0 (null) >Aug 01 11:05:00.218 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=10, @num_updates=0 >Aug 01 11:05:00.218 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_colocation rsc="virtualip" with-rsc="ag_cluster-clone" score="INFINITY" rsc-role="Started" with-rsc-role="Master" id="colocation-virtualip-ag_cluster-clone-INFINITY"/> >Aug 01 11:05:00.219 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=rhel91/cibadmin/2, version=0.10.0) >Aug 01 11:05:00.219 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 5 aborted by rsc_colocation.colocation-virtualip-ag_cluster-clone-INFINITY 'create': Configuration change | cib=0.10.0 source=te_update_diff_v2:465 path=/cib/configuration/constraints complete=true >Aug 01 11:05:00.219 rhel92 pacemaker-fenced [1478934] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create constraints >Aug 01 11:05:00.219 rhel92 pacemaker-fenced [1478934] (cib_devices_update) info: Updating devices to version 0.10.0 >Aug 01 11:05:00.219 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 11:05:00.223 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:05:00.223 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:05:00.223 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:05:00.223 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.223 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (set_instance_role) info: Choosing ag_cluster:2 (Promoted) on rhel91 for promotion >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (pcmk__set_instance_roles) info: ag_cluster-clone: Promoted 1 instances of a possible 1 >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:0 (Unpromoted rhel92) >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:1 (Unpromoted rhel93) >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:2 (Promoted rhel91) >Aug 01 11:05:00.224 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave virtualip (Started rhel91) >Aug 01 11:05:00.225 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-8.bz2 >Aug 01 11:05:00.232 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw >Aug 01 11:05:00.242 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:05:00.242 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 6 (ref=pe_calc-pacemaker-controld-1690913100-35) derived from /var/lib/pacemaker/pengine/pe-input-8.bz2 >Aug 01 11:05:00.242 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 6 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-8.bz2): Complete >Aug 01 11:05:00.242 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 11:05:00.242 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:05:00.248 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.10.0 of the CIB to disk (digest: b8bda43fbcea30c2cb48c799c34f743b) >Aug 01 11:05:00.255 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.fFq0Ey (digest: /var/lib/pacemaker/cib/cib.qX6s4t) >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:05:00 Monitor Caller is: monitor. >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:05:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:05:00 Replica is SECONDARY (2) >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:05:00 Instance name is rhel92. >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: 2023/08/01 11:05:00 Current master is . >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: lease_expiry after monitor update: >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180609" >Aug 01 11:05:00 ag(ag_cluster)[1480162]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: mssql_validate >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: mssql_monitor >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:01 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:01 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:01 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:01 Connected to the instance at localhost:1433 >Aug 01 11:05:05.220 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: fc5a67d41ccfdd65a9f2ae31b0b7bd5b for 0.10.0 (0x55c8f102f060 0) >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:06 Monitor Caller is: monitor. >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:06 Replica is SECONDARY (2) >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:06 Instance name is rhel92. >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: 2023/08/01 11:05:06 Current master is . >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: lease_expiry after monitor update: >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180609" >Aug 01 11:05:06 ag(ag_cluster)[1480280]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: mssql_validate >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: mssql_monitor >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:13 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:13 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:13 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:13 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:13 Connected to the instance at localhost:1433 >Aug 01 11:05:13.083 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180609 -> 20230801180626 | from rhel91 with no write delay >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:18 Monitor Caller is: monitor. >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:18 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:18 Replica is SECONDARY (2) >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:18 Instance name is rhel92. >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: 2023/08/01 11:05:18 Current master is . >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: lease_expiry after monitor update: >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180626" >Aug 01 11:05:18 ag(ag_cluster)[1480400]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: mssql_validate >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: mssql_monitor >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:18 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:18 Connected to the instance at localhost:1433 >Aug 01 11:05:20.022 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: --- 0.10.0 2 >Aug 01 11:05:20.022 rhel92 pacemaker-based [1478933] (cib_perform_op) info: Diff: +++ 0.11.0 (null) >Aug 01 11:05:20.022 rhel92 pacemaker-based [1478933] (cib_perform_op) info: + /cib: @epoch=11 >Aug 01 11:05:20.022 rhel92 pacemaker-based [1478933] (cib_perform_op) info: ++ /cib/configuration/constraints: <rsc_order first="ag_cluster-clone" then="virtualip" first-action="promote" then-action="start" id="order-ag_cluster-clone-virtualip-mandatory"/> >Aug 01 11:05:20.023 rhel92 pacemaker-based [1478933] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=rhel91/cibadmin/2, version=0.11.0) >Aug 01 11:05:20.023 rhel92 pacemaker-controld [1478938] (abort_transition_graph) info: Transition 6 aborted by rsc_order.order-ag_cluster-clone-virtualip-mandatory 'create': Configuration change | cib=0.11.0 source=te_update_diff_v2:465 path=/cib/configuration/constraints complete=true >Aug 01 11:05:20.023 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph >Aug 01 11:05:20.023 rhel92 pacemaker-fenced [1478934] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create constraints >Aug 01 11:05:20.023 rhel92 pacemaker-fenced [1478934] (cib_devices_update) info: Updating devices to version 0.11.0 >Aug 01 11:05:20.027 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel92 is online >Aug 01 11:05:20.027 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel93 is online >Aug 01 11:05:20.027 rhel92 pacemaker-schedulerd[1478937] (determine_online_status) info: rhel91 is online >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:0 active on rhel92 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:1 active on rhel93 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (remap_operation) info: Probe found ag_cluster:2 active and promoted on rhel91 at Aug 1 11:03:42 2023 >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (set_instance_role) info: Choosing ag_cluster:2 (Promoted) on rhel91 for promotion >Aug 01 11:05:20.028 rhel92 pacemaker-schedulerd[1478937] (pcmk__set_instance_roles) info: ag_cluster-clone: Promoted 1 instances of a possible 1 >Aug 01 11:05:20.029 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:0 (Unpromoted rhel92) >Aug 01 11:05:20.029 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:1 (Unpromoted rhel93) >Aug 01 11:05:20.029 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave ag_cluster:2 (Promoted rhel91) >Aug 01 11:05:20.029 rhel92 pacemaker-schedulerd[1478937] (rsc_action_default) info: Leave virtualip (Started rhel91) >Aug 01 11:05:20.029 rhel92 pacemaker-schedulerd[1478937] (pcmk__log_transition_summary) notice: Calculated transition 7, saving inputs in /var/lib/pacemaker/pengine/pe-input-9.bz2 >Aug 01 11:05:20.038 rhel92 pacemaker-based [1478933] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw >Aug 01 11:05:20.050 rhel92 pacemaker-controld [1478938] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_reply >Aug 01 11:05:20.050 rhel92 pacemaker-controld [1478938] (do_te_invoke) info: Processing graph 7 (ref=pe_calc-pacemaker-controld-1690913120-36) derived from /var/lib/pacemaker/pengine/pe-input-9.bz2 >Aug 01 11:05:20.050 rhel92 pacemaker-controld [1478938] (pcmk__execute_graph) notice: Transition 7 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-9.bz2): Complete >Aug 01 11:05:20.050 rhel92 pacemaker-controld [1478938] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd >Aug 01 11:05:20.050 rhel92 pacemaker-controld [1478938] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd >Aug 01 11:05:20.061 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Wrote version 0.11.0 of the CIB to disk (digest: 657045a818acc107d95d4d6ff834ee3c) >Aug 01 11:05:20.075 rhel92 pacemaker-based [1478933] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eTOdel (digest: /var/lib/pacemaker/cib/cib.6X5ilW) >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:23 Monitor Caller is: monitor. >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:23 Replica is SECONDARY (2) >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:23 Instance name is rhel92. >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: 2023/08/01 11:05:23 Current master is . >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: lease_expiry after monitor update: >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180626" >Aug 01 11:05:23 ag(ag_cluster)[1480516]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:25.024 rhel92 pacemaker-based [1478933] (cib_process_ping) info: Reporting our current digest to rhel92: 4f446d4cd5b0ecfc3476963d20d96b60 for 0.11.0 (0x55c8f102f340 0) >Aug 01 11:05:29.213 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180626 -> 20230801180642 | from rhel91 with no write delay >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: mssql_validate >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: mssql_monitor >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:30 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:30 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:30 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:30 Connected to the instance at localhost:1433 >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:35 Monitor Caller is: monitor. >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:35 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:35 Replica is SECONDARY (2) >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:35 Instance name is rhel92. >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: 2023/08/01 11:05:35 Current master is . >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: lease_expiry after monitor update: >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180642" >Aug 01 11:05:35 ag(ag_cluster)[1480635]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: mssql_validate >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: mssql_monitor >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:35 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:35 Connected to the instance at localhost:1433 >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:40 Monitor Caller is: monitor. >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:40 Replica is SECONDARY (2) >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:40 Instance name is rhel92. >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: 2023/08/01 11:05:40 Current master is . >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: lease_expiry after monitor update: >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180642" >Aug 01 11:05:40 ag(ag_cluster)[1480753]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:45.341 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180642 -> 20230801180658 | from rhel91 with no write delay >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: mssql_validate >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: mssql_monitor >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:47 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:47 Connected to the instance at localhost:1433 >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:52 Monitor Caller is: monitor. >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:52 Replica is SECONDARY (2) >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:52 Instance name is rhel92. >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: 2023/08/01 11:05:52 Current master is . >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: lease_expiry after monitor update: >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180658" >Aug 01 11:05:52 ag(ag_cluster)[1480870]: INFO: ag_cluster monitor : 0 >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: mssql_validate >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: Resource agent invoked with: monitor >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: mssql_monitor >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:05:52 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:52 Connected to the instance at localhost:1433 >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:57 Monitor Caller is: monitor. >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:57 Replica is SECONDARY (2) >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:57 Instance name is rhel92. >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: 2023/08/01 11:05:57 Current master is . >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: lease_expiry after monitor update: >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180658" >Aug 01 11:05:57 ag(ag_cluster)[1481006]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:01.467 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180658 -> 20230801180714 | from rhel91 with no write delay >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: mssql_validate >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: mssql_monitor >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:04 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:04 Connected to the instance at localhost:1433 >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:09 Monitor Caller is: monitor. >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:09 Replica is SECONDARY (2) >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:09 Instance name is rhel92. >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: 2023/08/01 11:06:09 Current master is . >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: lease_expiry after monitor update: >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180714" >Aug 01 11:06:09 ag(ag_cluster)[1481124]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: mssql_validate >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: mssql_monitor >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:09 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:09 Connected to the instance at localhost:1433 >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:14 Monitor Caller is: monitor. >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:14 Replica is SECONDARY (2) >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:14 Instance name is rhel92. >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: 2023/08/01 11:06:14 Current master is . >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: lease_expiry after monitor update: >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180714" >Aug 01 11:06:14 ag(ag_cluster)[1481244]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:17.591 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180714 -> 20230801180730 | from rhel91 with no write delay >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: mssql_validate >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: mssql_monitor >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:21 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:21 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:21 Connected to the instance at localhost:1433 >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:26 Monitor Caller is: monitor. >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:26 Replica is SECONDARY (2) >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:26 Instance name is rhel92. >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: 2023/08/01 11:06:26 Current master is . >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: lease_expiry after monitor update: >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180730" >Aug 01 11:06:26 ag(ag_cluster)[1481361]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: mssql_validate >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: mssql_monitor >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:26 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:26 Connected to the instance at localhost:1433 >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:31 Monitor Caller is: monitor. >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:31 Replica is SECONDARY (2) >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:31 Instance name is rhel92. >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: 2023/08/01 11:06:31 Current master is . >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: lease_expiry after monitor update: >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180730" >Aug 01 11:06:31 ag(ag_cluster)[1481479]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:33.736 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180730 -> 20230801180746 | from rhel91 with no write delay >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: mssql_validate >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: mssql_monitor >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:38 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:38 Connected to the instance at localhost:1433 >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:43 Monitor Caller is: monitor. >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:43 Replica is SECONDARY (2) >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:43 Instance name is rhel92. >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: 2023/08/01 11:06:43 Current master is . >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: lease_expiry after monitor update: >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180746" >Aug 01 11:06:43 ag(ag_cluster)[1481672]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: mssql_validate >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: mssql_monitor >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:43 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:43 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:43 Connected to the instance at localhost:1433 >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:48 Monitor Caller is: monitor. >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:48 Replica is SECONDARY (2) >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:48 Instance name is rhel92. >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: 2023/08/01 11:06:48 Current master is . >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: lease_expiry after monitor update: >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180746" >Aug 01 11:06:48 ag(ag_cluster)[1481789]: INFO: ag_cluster monitor : 0 >Aug 01 11:06:49.884 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180746 -> 20230801180802 | from rhel91 with no write delay >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: mssql_validate >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: Resource agent invoked with: monitor >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: mssql_monitor >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:06:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:06:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:06:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:06:55 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:06:55 Connected to the instance at localhost:1433 >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:07:00 Monitor Caller is: monitor. >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:07:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:07:00 Replica is SECONDARY (2) >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:07:00 Instance name is rhel92. >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: 2023/08/01 11:07:00 Current master is . >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: lease_expiry after monitor update: >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180802" >Aug 01 11:07:00 ag(ag_cluster)[1481926]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:00 ag(ag_cluster)[1482101]: INFO: mssql_validate >Aug 01 11:07:00 ag(ag_cluster)[1482101]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:07:00 ag(ag_cluster)[1482101]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:00 ag(ag_cluster)[1482101]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:00 ag(ag_cluster)[1482101]: INFO: mssql_monitor >Aug 01 11:07:01 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:01 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:01 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:07:01 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:01 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:01 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:01 Connected to the instance at localhost:1433 >Aug 01 11:07:06.044 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180802 -> 20230801180819 | from rhel91 with no write delay >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:06 Monitor Caller is: monitor. >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:06 Replica is SECONDARY (2) >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:06 Instance name is rhel92. >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: 2023/08/01 11:07:06 Current master is . >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: lease_expiry after monitor update: >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180819" >Aug 01 11:07:06 ag(ag_cluster)[1482101]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:12 ag(ag_cluster)[1482220]: INFO: mssql_validate >Aug 01 11:07:12 ag(ag_cluster)[1482220]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:07:12 ag(ag_cluster)[1482220]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:12 ag(ag_cluster)[1482220]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:12 ag(ag_cluster)[1482220]: INFO: mssql_monitor >Aug 01 11:07:13 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:13 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:13 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:13 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:07:13 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:13 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:13 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:13 Connected to the instance at localhost:1433 >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:18 Monitor Caller is: monitor. >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:18 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:18 Replica is SECONDARY (2) >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:18 Instance name is rhel92. >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: 2023/08/01 11:07:18 Current master is . >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: lease_expiry after monitor update: >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180819" >Aug 01 11:07:18 ag(ag_cluster)[1482220]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: mssql_validate >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: mssql_monitor >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:18 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:18 Connected to the instance at localhost:1433 >Aug 01 11:07:22.202 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180819 -> 20230801180835 | from rhel91 with no write delay >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:23 Monitor Caller is: monitor. >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:23 Replica is SECONDARY (2) >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:23 Instance name is rhel92. >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: 2023/08/01 11:07:23 Current master is . >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: lease_expiry after monitor update: >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180835" >Aug 01 11:07:23 ag(ag_cluster)[1482337]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: mssql_validate >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: mssql_monitor >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:30 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:30 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:30 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:30 Connected to the instance at localhost:1433 >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:35 Monitor Caller is: monitor. >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:35 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:35 Replica is SECONDARY (2) >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:35 Instance name is rhel92. >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: 2023/08/01 11:07:35 Current master is . >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: lease_expiry after monitor update: >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180835" >Aug 01 11:07:35 ag(ag_cluster)[1482454]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: mssql_validate >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: mssql_monitor >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:35 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:35 Connected to the instance at localhost:1433 >Aug 01 11:07:38.349 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180835 -> 20230801180851 | from rhel91 with no write delay >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:40 Monitor Caller is: monitor. >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:40 Replica is SECONDARY (2) >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:40 Instance name is rhel92. >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: 2023/08/01 11:07:40 Current master is . >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: lease_expiry after monitor update: >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180851" >Aug 01 11:07:40 ag(ag_cluster)[1482572]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: mssql_validate >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: mssql_monitor >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:47 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:47 Connected to the instance at localhost:1433 >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:52 Monitor Caller is: monitor. >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:52 Replica is SECONDARY (2) >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:52 Instance name is rhel92. >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: 2023/08/01 11:07:52 Current master is . >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: lease_expiry after monitor update: >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180851" >Aug 01 11:07:52 ag(ag_cluster)[1482689]: INFO: ag_cluster monitor : 0 >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: mssql_validate >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: Resource agent invoked with: monitor >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: mssql_monitor >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:07:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:07:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:07:52 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:07:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:07:54.473 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180851 -> 20230801180907 | from rhel91 with no write delay >Aug 01 11:07:57 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:07:57 Connected to the instance at localhost:1433 >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:08:02 Monitor Caller is: monitor. >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:08:02 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:08:02 Replica is SECONDARY (2) >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:08:02 Instance name is rhel92. >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: 2023/08/01 11:08:02 Current master is . >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: lease_expiry after monitor update: >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180907" >Aug 01 11:08:02 ag(ag_cluster)[1482806]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: mssql_validate >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: mssql_monitor >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:04 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:04 Connected to the instance at localhost:1433 >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:09 Monitor Caller is: monitor. >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:09 Replica is SECONDARY (2) >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:09 Instance name is rhel92. >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: 2023/08/01 11:08:09 Current master is . >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: lease_expiry after monitor update: >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180907" >Aug 01 11:08:09 ag(ag_cluster)[1482924]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:10.622 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180907 -> 20230801180923 | from rhel91 with no write delay >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: mssql_validate >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: mssql_monitor >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:12 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:12 Connected to the instance at localhost:1433 >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:17 Monitor Caller is: monitor. >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:17 Replica is SECONDARY (2) >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:17 Instance name is rhel92. >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: 2023/08/01 11:08:17 Current master is . >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: lease_expiry after monitor update: >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180923" >Aug 01 11:08:17 ag(ag_cluster)[1483043]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: mssql_validate >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: mssql_monitor >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:21 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:21 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:21 Connected to the instance at localhost:1433 >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:26 Monitor Caller is: monitor. >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:26 Replica is SECONDARY (2) >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:26 Instance name is rhel92. >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: 2023/08/01 11:08:26 Current master is . >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: lease_expiry after monitor update: >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180923" >Aug 01 11:08:26 ag(ag_cluster)[1483160]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:26.750 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180923 -> 20230801180939 | from rhel91 with no write delay >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: mssql_validate >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: mssql_monitor >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:27 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:27 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:27 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:27 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:27 Connected to the instance at localhost:1433 >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:32 Monitor Caller is: monitor. >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:32 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:32 Replica is SECONDARY (2) >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:32 Instance name is rhel92. >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: 2023/08/01 11:08:32 Current master is . >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: lease_expiry after monitor update: >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180939" >Aug 01 11:08:32 ag(ag_cluster)[1483278]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: mssql_validate >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: mssql_monitor >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:38 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:38 Connected to the instance at localhost:1433 >Aug 01 11:08:42.900 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180939 -> 20230801180955 | from rhel91 with no write delay >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:43 Monitor Caller is: monitor. >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:43 Replica is SECONDARY (2) >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:43 Instance name is rhel92. >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: 2023/08/01 11:08:43 Current master is . >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: lease_expiry after monitor update: >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180955" >Aug 01 11:08:43 ag(ag_cluster)[1483397]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: mssql_validate >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: mssql_monitor >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:43 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:43 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:43 Connected to the instance at localhost:1433 >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:48 Monitor Caller is: monitor. >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:48 Replica is SECONDARY (2) >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:48 Instance name is rhel92. >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: 2023/08/01 11:08:48 Current master is . >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: lease_expiry after monitor update: >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801180955" >Aug 01 11:08:48 ag(ag_cluster)[1483514]: INFO: ag_cluster monitor : 0 >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: mssql_validate >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: Resource agent invoked with: monitor >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: mssql_monitor >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:08:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:08:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:08:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:08:55 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:08:55 Connected to the instance at localhost:1433 >Aug 01 11:08:59.051 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801180955 -> 20230801181012 | from rhel91 with no write delay >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:09:00 Monitor Caller is: monitor. >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:09:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:09:00 Replica is SECONDARY (2) >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:09:00 Instance name is rhel92. >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: 2023/08/01 11:09:00 Current master is . >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: lease_expiry after monitor update: >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181012" >Aug 01 11:09:00 ag(ag_cluster)[1483631]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:00 ag(ag_cluster)[1483750]: INFO: mssql_validate >Aug 01 11:09:00 ag(ag_cluster)[1483750]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:09:00 ag(ag_cluster)[1483750]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:00 ag(ag_cluster)[1483750]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:00 ag(ag_cluster)[1483750]: INFO: mssql_monitor >Aug 01 11:09:01 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:00 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:01 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:00 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:09:01 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:00 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:01 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:01 Connected to the instance at localhost:1433 >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:06 Monitor Caller is: monitor. >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:06 Replica is SECONDARY (2) >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:06 Instance name is rhel92. >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: 2023/08/01 11:09:06 Current master is . >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: lease_expiry after monitor update: >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181012" >Aug 01 11:09:06 ag(ag_cluster)[1483750]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:12 ag(ag_cluster)[1483869]: INFO: mssql_validate >Aug 01 11:09:12 ag(ag_cluster)[1483869]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:09:12 ag(ag_cluster)[1483869]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:12 ag(ag_cluster)[1483869]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:12 ag(ag_cluster)[1483869]: INFO: mssql_monitor >Aug 01 11:09:13 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:13 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:09:13 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:13 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:13 Connected to the instance at localhost:1433 >Aug 01 11:09:15.179 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181012 -> 20230801181028 | from rhel91 with no write delay >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:18 Monitor Caller is: monitor. >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:18 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:18 Replica is SECONDARY (2) >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:18 Instance name is rhel92. >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: 2023/08/01 11:09:18 Current master is . >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: lease_expiry after monitor update: >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181028" >Aug 01 11:09:18 ag(ag_cluster)[1483869]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: mssql_validate >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: mssql_monitor >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:18 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:18 Connected to the instance at localhost:1433 >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:23 Monitor Caller is: monitor. >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:23 Replica is SECONDARY (2) >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:23 Instance name is rhel92. >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: 2023/08/01 11:09:23 Current master is . >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: lease_expiry after monitor update: >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181028" >Aug 01 11:09:23 ag(ag_cluster)[1483987]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: mssql_validate >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: mssql_monitor >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:30 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:30 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:30 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:30 Connected to the instance at localhost:1433 >Aug 01 11:09:31.305 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181028 -> 20230801181044 | from rhel91 with no write delay >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:35 Monitor Caller is: monitor. >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:35 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:35 Replica is SECONDARY (2) >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:35 Instance name is rhel92. >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: 2023/08/01 11:09:35 Current master is . >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: lease_expiry after monitor update: >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181044" >Aug 01 11:09:35 ag(ag_cluster)[1484105]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: mssql_validate >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: mssql_monitor >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:35 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:35 Connected to the instance at localhost:1433 >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:40 Monitor Caller is: monitor. >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:40 Replica is SECONDARY (2) >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:40 Instance name is rhel92. >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: 2023/08/01 11:09:40 Current master is . >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: lease_expiry after monitor update: >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181044" >Aug 01 11:09:40 ag(ag_cluster)[1484223]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: mssql_validate >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: mssql_monitor >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:47 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:47 Connected to the instance at localhost:1433 >Aug 01 11:09:47.429 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181044 -> 20230801181100 | from rhel91 with no write delay >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:52 Monitor Caller is: monitor. >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:52 Replica is SECONDARY (2) >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:52 Instance name is rhel92. >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: 2023/08/01 11:09:52 Current master is . >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: lease_expiry after monitor update: >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181100" >Aug 01 11:09:52 ag(ag_cluster)[1484347]: INFO: ag_cluster monitor : 0 >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: mssql_validate >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: Resource agent invoked with: monitor >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: mssql_monitor >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:09:52 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:52 Connected to the instance at localhost:1433 >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:57 Monitor Caller is: monitor. >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:57 Replica is SECONDARY (2) >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:57 Instance name is rhel92. >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: 2023/08/01 11:09:57 Current master is . >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: lease_expiry after monitor update: >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181100" >Aug 01 11:09:57 ag(ag_cluster)[1484464]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:03.552 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181100 -> 20230801181116 | from rhel91 with no write delay >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: mssql_validate >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: mssql_monitor >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:04 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:04 Connected to the instance at localhost:1433 >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:09 Monitor Caller is: monitor. >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:09 Replica is SECONDARY (2) >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:09 Instance name is rhel92. >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: 2023/08/01 11:10:09 Current master is . >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: lease_expiry after monitor update: >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181116" >Aug 01 11:10:09 ag(ag_cluster)[1484581]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: mssql_validate >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: mssql_monitor >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:09 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:09 Connected to the instance at localhost:1433 >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:14 Monitor Caller is: monitor. >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:14 Replica is SECONDARY (2) >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:14 Instance name is rhel92. >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: 2023/08/01 11:10:14 Current master is . >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: lease_expiry after monitor update: >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181116" >Aug 01 11:10:14 ag(ag_cluster)[1484699]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:19.686 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181116 -> 20230801181132 | from rhel91 with no write delay >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: mssql_validate >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: mssql_monitor >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:21 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:21 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:21 Connected to the instance at localhost:1433 >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:26 Monitor Caller is: monitor. >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:26 Replica is SECONDARY (2) >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:26 Instance name is rhel92. >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: 2023/08/01 11:10:26 Current master is . >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: lease_expiry after monitor update: >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181132" >Aug 01 11:10:26 ag(ag_cluster)[1484817]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: mssql_validate >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: mssql_monitor >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:26 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:26 Connected to the instance at localhost:1433 >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:31 Monitor Caller is: monitor. >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:31 Replica is SECONDARY (2) >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:31 Instance name is rhel92. >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: 2023/08/01 11:10:31 Current master is . >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: lease_expiry after monitor update: >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181132" >Aug 01 11:10:31 ag(ag_cluster)[1484934]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:35.835 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181132 -> 20230801181148 | from rhel91 with no write delay >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: mssql_validate >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: mssql_monitor >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:38 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:38 Connected to the instance at localhost:1433 >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:43 Monitor Caller is: monitor. >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:43 Replica is SECONDARY (2) >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:43 Instance name is rhel92. >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: 2023/08/01 11:10:43 Current master is . >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: lease_expiry after monitor update: >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181148" >Aug 01 11:10:43 ag(ag_cluster)[1485053]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: mssql_validate >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: mssql_monitor >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:43 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:43 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:43 Connected to the instance at localhost:1433 >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:48 Monitor Caller is: monitor. >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:48 Replica is SECONDARY (2) >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:48 Instance name is rhel92. >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: 2023/08/01 11:10:48 Current master is . >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: lease_expiry after monitor update: >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181148" >Aug 01 11:10:48 ag(ag_cluster)[1485170]: INFO: ag_cluster monitor : 0 >Aug 01 11:10:51.984 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181148 -> 20230801181204 | from rhel91 with no write delay >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: mssql_validate >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: Resource agent invoked with: monitor >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: mssql_monitor >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:10:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:10:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:10:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:10:55 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:10:55 Connected to the instance at localhost:1433 >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:11:00 Monitor Caller is: monitor. >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:11:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:11:00 Replica is SECONDARY (2) >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:11:00 Instance name is rhel92. >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: 2023/08/01 11:11:00 Current master is . >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: lease_expiry after monitor update: >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181204" >Aug 01 11:11:00 ag(ag_cluster)[1485287]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: mssql_validate >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: mssql_monitor >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:00 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:00 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:00 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:00 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:00 Connected to the instance at localhost:1433 >Aug 01 11:11:05 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:05 Monitor Caller is: monitor. >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:05 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:05 Replica is SECONDARY (2) >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:05 Instance name is rhel92. >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: monitor: 2023/08/01 11:11:05 Current master is . >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: lease_expiry after monitor update: >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181204" >Aug 01 11:11:06 ag(ag_cluster)[1485405]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:08.108 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181204 -> 20230801181221 | from rhel91 with no write delay >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: mssql_validate >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: mssql_monitor >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:12 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:12 Connected to the instance at localhost:1433 >Aug 01 11:11:17 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:17 Monitor Caller is: monitor. >Aug 01 11:11:17 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:17 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:17 Replica is SECONDARY (2) >Aug 01 11:11:17 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:17 Instance name is rhel92. >Aug 01 11:11:18 ag(ag_cluster)[1485523]: INFO: monitor: 2023/08/01 11:11:17 Current master is . >Aug 01 11:11:18 ag(ag_cluster)[1485523]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:18 ag(ag_cluster)[1485523]: INFO: lease_expiry after monitor update: >Aug 01 11:11:18 ag(ag_cluster)[1485523]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181221" >Aug 01 11:11:18 ag(ag_cluster)[1485523]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: mssql_validate >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: mssql_monitor >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:18 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:18 Connected to the instance at localhost:1433 >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:23 Monitor Caller is: monitor. >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:23 Replica is SECONDARY (2) >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:23 Instance name is rhel92. >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: 2023/08/01 11:11:23 Current master is . >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: lease_expiry after monitor update: >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181221" >Aug 01 11:11:23 ag(ag_cluster)[1485639]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:24.231 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181221 -> 20230801181237 | from rhel91 with no write delay >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: mssql_validate >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: mssql_monitor >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:30 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:30 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:30 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:30 Connected to the instance at localhost:1433 >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:35 Monitor Caller is: monitor. >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:35 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:35 Replica is SECONDARY (2) >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:35 Instance name is rhel92. >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: 2023/08/01 11:11:35 Current master is . >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: lease_expiry after monitor update: >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181237" >Aug 01 11:11:35 ag(ag_cluster)[1485757]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: mssql_validate >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: mssql_monitor >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:35 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:35 Connected to the instance at localhost:1433 >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:40 Monitor Caller is: monitor. >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:40 Replica is SECONDARY (2) >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:40 Instance name is rhel92. >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: 2023/08/01 11:11:40 Current master is . >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: lease_expiry after monitor update: >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181237" >Aug 01 11:11:40 ag(ag_cluster)[1485875]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:40.363 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181237 -> 20230801181253 | from rhel91 with no write delay >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: mssql_validate >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: mssql_monitor >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:47 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:47 Connected to the instance at localhost:1433 >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:52 Monitor Caller is: monitor. >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:52 Replica is SECONDARY (2) >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:52 Instance name is rhel92. >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: 2023/08/01 11:11:52 Current master is . >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: lease_expiry after monitor update: >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181253" >Aug 01 11:11:52 ag(ag_cluster)[1485996]: INFO: ag_cluster monitor : 0 >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: mssql_validate >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: Resource agent invoked with: monitor >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: mssql_monitor >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:11:52 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:52 Connected to the instance at localhost:1433 >Aug 01 11:11:56.489 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181253 -> 20230801181309 | from rhel91 with no write delay >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:57 Monitor Caller is: monitor. >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:57 Replica is SECONDARY (2) >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:57 Instance name is rhel92. >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: 2023/08/01 11:11:57 Current master is . >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: lease_expiry after monitor update: >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181309" >Aug 01 11:11:57 ag(ag_cluster)[1486112]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: mssql_validate >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: mssql_monitor >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:04 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:04 Connected to the instance at localhost:1433 >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:09 Monitor Caller is: monitor. >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:09 Replica is SECONDARY (2) >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:09 Instance name is rhel92. >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: 2023/08/01 11:12:09 Current master is . >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: lease_expiry after monitor update: >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181309" >Aug 01 11:12:09 ag(ag_cluster)[1486230]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: mssql_validate >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: mssql_monitor >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:09 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:09 Connected to the instance at localhost:1433 >Aug 01 11:12:12.613 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181309 -> 20230801181325 | from rhel91 with no write delay >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:14 Monitor Caller is: monitor. >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:14 Replica is SECONDARY (2) >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:14 Instance name is rhel92. >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: 2023/08/01 11:12:14 Current master is . >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: lease_expiry after monitor update: >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181325" >Aug 01 11:12:14 ag(ag_cluster)[1486348]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: mssql_validate >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: mssql_monitor >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:21 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:21 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:21 Connected to the instance at localhost:1433 >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:26 Monitor Caller is: monitor. >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:26 Replica is SECONDARY (2) >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:26 Instance name is rhel92. >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: 2023/08/01 11:12:26 Current master is . >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: lease_expiry after monitor update: >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181325" >Aug 01 11:12:26 ag(ag_cluster)[1486465]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: mssql_validate >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: mssql_monitor >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:26 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:26 Connected to the instance at localhost:1433 >Aug 01 11:12:28.758 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181325 -> 20230801181341 | from rhel91 with no write delay >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:31 Monitor Caller is: monitor. >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:31 Replica is SECONDARY (2) >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:31 Instance name is rhel92. >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: 2023/08/01 11:12:31 Current master is . >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: lease_expiry after monitor update: >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181341" >Aug 01 11:12:31 ag(ag_cluster)[1486582]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: mssql_validate >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: mssql_monitor >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:38 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:38 Connected to the instance at localhost:1433 >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:43 Monitor Caller is: monitor. >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:43 Replica is SECONDARY (2) >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:43 Instance name is rhel92. >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: 2023/08/01 11:12:43 Current master is . >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: lease_expiry after monitor update: >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181341" >Aug 01 11:12:43 ag(ag_cluster)[1486700]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: mssql_validate >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: mssql_monitor >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:43 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:43 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:43 Connected to the instance at localhost:1433 >Aug 01 11:12:44.902 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181341 -> 20230801181357 | from rhel91 with no write delay >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:48 Monitor Caller is: monitor. >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:48 Replica is SECONDARY (2) >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:48 Instance name is rhel92. >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: 2023/08/01 11:12:48 Current master is . >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: lease_expiry after monitor update: >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181357" >Aug 01 11:12:48 ag(ag_cluster)[1486817]: INFO: ag_cluster monitor : 0 >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: mssql_validate >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: Resource agent invoked with: monitor >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: mssql_monitor >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:12:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:12:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:12:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:12:55 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:12:55 Connected to the instance at localhost:1433 >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:13:00 Monitor Caller is: monitor. >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:13:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:13:00 Replica is SECONDARY (2) >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:13:00 Instance name is rhel92. >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: 2023/08/01 11:13:00 Current master is . >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: lease_expiry after monitor update: >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181357" >Aug 01 11:13:00 ag(ag_cluster)[1486934]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: mssql_validate >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: mssql_monitor >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:00 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:00 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:00 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:00 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:00 Connected to the instance at localhost:1433 >Aug 01 11:13:01.030 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181357 -> 20230801181414 | from rhel91 with no write delay >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:05 Monitor Caller is: monitor. >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:05 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:05 Replica is SECONDARY (2) >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:05 Instance name is rhel92. >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: 2023/08/01 11:13:05 Current master is . >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: lease_expiry after monitor update: >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181414" >Aug 01 11:13:05 ag(ag_cluster)[1487051]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: mssql_validate >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: mssql_monitor >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:12 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:12 Connected to the instance at localhost:1433 >Aug 01 11:13:17.157 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181414 -> 20230801181430 | from rhel91 with no write delay >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:17 Monitor Caller is: monitor. >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:17 Replica is SECONDARY (2) >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:17 Instance name is rhel92. >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: 2023/08/01 11:13:17 Current master is . >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: lease_expiry after monitor update: >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181430" >Aug 01 11:13:17 ag(ag_cluster)[1487170]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: mssql_validate >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: mssql_monitor >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:13:17 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:17 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:18 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:18 Connected to the instance at localhost:1433 >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:23 Monitor Caller is: monitor. >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:23 Replica is SECONDARY (2) >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:23 Instance name is rhel92. >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: 2023/08/01 11:13:23 Current master is . >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: lease_expiry after monitor update: >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181430" >Aug 01 11:13:23 ag(ag_cluster)[1487287]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: mssql_validate >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: mssql_monitor >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:13:29 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:29 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:30 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:30 Connected to the instance at localhost:1433 >Aug 01 11:13:33.285 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181430 -> 20230801181446 | from rhel91 with no write delay >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:35 Monitor Caller is: monitor. >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:35 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:35 Replica is SECONDARY (2) >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:35 Instance name is rhel92. >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: 2023/08/01 11:13:35 Current master is . >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: lease_expiry after monitor update: >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181446" >Aug 01 11:13:35 ag(ag_cluster)[1487405]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: mssql_validate >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: mssql_monitor >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:35 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:35 Connected to the instance at localhost:1433 >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:40 Monitor Caller is: monitor. >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:40 Replica is SECONDARY (2) >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:40 Instance name is rhel92. >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: 2023/08/01 11:13:40 Current master is . >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: lease_expiry after monitor update: >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181446" >Aug 01 11:13:40 ag(ag_cluster)[1487521]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: mssql_validate >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: mssql_monitor >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:47 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:47 Connected to the instance at localhost:1433 >Aug 01 11:13:49.413 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181446 -> 20230801181502 | from rhel91 with no write delay >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:52 Monitor Caller is: monitor. >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:52 Replica is SECONDARY (2) >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:52 Instance name is rhel92. >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: 2023/08/01 11:13:52 Current master is . >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: lease_expiry after monitor update: >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181502" >Aug 01 11:13:52 ag(ag_cluster)[1487640]: INFO: ag_cluster monitor : 0 >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: mssql_validate >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: Resource agent invoked with: monitor >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: mssql_monitor >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:13:52 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:52 Connected to the instance at localhost:1433 >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:57 Monitor Caller is: monitor. >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:57 Replica is SECONDARY (2) >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:57 Instance name is rhel92. >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: 2023/08/01 11:13:57 Current master is . >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: lease_expiry after monitor update: >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181502" >Aug 01 11:13:57 ag(ag_cluster)[1487756]: INFO: ag_cluster monitor : 0 >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: mssql_validate >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: Resource agent invoked with: monitor >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: mssql_monitor >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:14:04 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:04 Connected to the instance at localhost:1433 >Aug 01 11:14:05.540 rhel92 pacemaker-attrd [1478936] (update_attr_on_host) notice: Setting ag_cluster-lease-expiry[rhel91]: 20230801181502 -> 20230801181518 | from rhel91 with no write delay >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:09 Monitor Caller is: monitor. >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:09 Replica is SECONDARY (2) >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:09 Instance name is rhel92. >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: 2023/08/01 11:14:09 Current master is . >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: lease_expiry after monitor update: >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181518" >Aug 01 11:14:09 ag(ag_cluster)[1487874]: INFO: ag_cluster monitor : 0 >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: mssql_validate >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: Resource agent invoked with: monitor >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: mssql_monitor >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Aug 01 11:14:09 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:09 Connected to the instance at localhost:1433 >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:14 Monitor Caller is: monitor. >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance rhel92 >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:14 Replica is SECONDARY (2) >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:14 Instance name is rhel92. >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: 2023/08/01 11:14:14 Current master is . >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: monitor: PROMOTION_SCORE: 10 >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: lease_expiry after monitor update: >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="rhel92" value="" >name="ag_cluster-lease-expiry" host="rhel93" value="" >name="ag_cluster-lease-expiry" host="rhel91" value="20230801181518" >Aug 01 11:14:14 ag(ag_cluster)[1487992]: INFO: ag_cluster monitor : 0
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 2224249
:
1976688
|
1976689
|
1980299
|
1980300
|
1981341
| 1981342 |
1981343