Login
Log in using an SSO provider:
Fedora Account System
Red Hat Associate
Red Hat Customer
Login using a Red Hat Bugzilla account
Forgot Password
Create an Account
Red Hat Bugzilla – Attachment 1976685 Details for
Bug 2221772
Automatic failover - Pacemaker 2.0 , on-fail=demote with sql server 3node replicas, stat-failure-is-fatal not being adhered to
Home
New
Search
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh89 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
[?]
This site requires JavaScript to be enabled to function correctly, please enable it.
pacemaker logs_updated
pacemaker2_7_19.log (text/plain), 440.86 KB, created by
Aravind Mahadevan
on 2023-07-20 09:09:02 UTC
(
hide
)
Description:
pacemaker logs_updated
Filename:
MIME Type:
Creator:
Aravind Mahadevan
Created:
2023-07-20 09:09:02 UTC
Size:
440.86 KB
patch
obsolete
>Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: 2023/07/19 14:37:56 Monitor Caller is: monitor. >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: 2023/07/19 14:37:56 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: 2023/07/19 14:37:56 Replica is SECONDARY (2) >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: 2023/07/19 14:37:56 Instance name is server2. >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: 2023/07/19 14:37:56 Current master is server1. >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: monitor: PROMOTION_SCORE: 10 >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: lease_expiry after monitor update: >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:37:56 ag(ag_cluster)[38997]: INFO: ag_cluster monitor : 0 >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: mssql_validate >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: Resource agent invoked with: monitor >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: mssql_monitor >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:03 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:03 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:03 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:03 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:03 Connected to the instance at localhost:1433 >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:08 Monitor Caller is: monitor. >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:08 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:08 Replica is SECONDARY (2) >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:08 Instance name is server2. >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: 2023/07/19 14:38:08 Current master is server1. >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: monitor: PROMOTION_SCORE: 10 >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: lease_expiry after monitor update: >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:38:08 ag(ag_cluster)[39152]: INFO: ag_cluster monitor : 0 >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: mssql_validate >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: Resource agent invoked with: monitor >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: mssql_monitor >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:08 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:08 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:08 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:08 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:08 Connected to the instance at localhost:1433 >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:13 Monitor Caller is: monitor. >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:13 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:13 Replica is SECONDARY (2) >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:13 Instance name is server2. >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: 2023/07/19 14:38:13 Current master is server1. >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: monitor: PROMOTION_SCORE: 10 >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: lease_expiry after monitor update: >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:38:13 ag(ag_cluster)[39306]: INFO: ag_cluster monitor : 0 >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: mssql_validate >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: OCF_RESKEY_CRM_meta_interval value: 12000 >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: Resource agent invoked with: monitor >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: mssql_monitor >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60] >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:20 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:20 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:20 Connected to the instance at localhost:1433 >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:25 Monitor Caller is: monitor. >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:25 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:25 Replica is SECONDARY (2) >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:25 Instance name is server2. >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: 2023/07/19 14:38:25 Current master is server1. >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: monitor: PROMOTION_SCORE: 10 >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: lease_expiry after monitor update: >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:38:25 ag(ag_cluster)[39460]: INFO: ag_cluster monitor : 0 >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: mssql_validate >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: Resource agent invoked with: notify >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: mssql_notify pre-demote >Jul 19 14:38:25 ag(ag_cluster)[39614]: INFO: ag_cluster notify : 0 >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: mssql_validate >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: OCF_RESKEY_CRM_meta_interval value: 10000 >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: Resource agent invoked with: monitor >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: mssql_monitor >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:25 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60] >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:25 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:25 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:25 Connected to the instance at localhost:1433 >Jul 19 14:38:30 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:30 Monitor Caller is: monitor. >Jul 19 14:38:30 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:30 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:30 Replica is SECONDARY (2) >Jul 19 14:38:30 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:30 Instance name is server2. >Jul 19 14:38:31 ag(ag_cluster)[39659]: INFO: monitor: 2023/07/19 14:38:30 Current master is server1. >Jul 19 14:38:31 ag(ag_cluster)[39659]: INFO: monitor: PROMOTION_SCORE: 10 >Jul 19 14:38:31 ag(ag_cluster)[39659]: INFO: lease_expiry after monitor update: >Jul 19 14:38:31 ag(ag_cluster)[39659]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:38:31 ag(ag_cluster)[39659]: INFO: ag_cluster monitor : 0 >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: mssql_validate >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: Resource agent invoked with: notify >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: mssql_notify post-demote >Jul 19 14:38:31 ag(ag_cluster)[39813]: INFO: ag_cluster notify : 0 >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: mssql_validate >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: Resource agent invoked with: notify >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: mssql_notify pre-stop >Jul 19 14:38:31 ag(ag_cluster)[39858]: INFO: ag_cluster notify : 0 >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: mssql_validate >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: Resource agent invoked with: notify >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: mssql_notify post-stop >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: notify: 2023/07/19 14:38:31 Replica is SECONDARY (2) >Jul 19 14:38:31 ag(ag_cluster)[39903]: INFO: ag_cluster notify : 0 >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: mssql_validate >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: Resource agent invoked with: notify >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: mssql_notify pre-start >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: 2023/07/19 14:38:31 Replica is SECONDARY (2) >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: 2023/07/19 14:38:31 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: 2023/07/19 14:38:31 Sequence number is 3847:F (61877593833487) >Jul 19 14:38:31 ag(ag_cluster)[39977]: INFO: notify: SEQUENCE_NUMBER: 61877593833487 >Jul 19 14:38:36 ag(ag_cluster)[39977]: INFO: ag_cluster notify : 0 >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: mssql_validate >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: Resource agent invoked with: notify >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: mssql_notify pre-stop >Jul 19 14:38:36 ag(ag_cluster)[40076]: INFO: ag_cluster notify : 0 >Jul 19 14:38:36 ag(ag_cluster)[40121]: INFO: mssql_validate >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: Resource agent invoked with: stop >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: mssql_stop >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: stop: 2023/07/19 14:38:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: stop: 2023/07/19 14:38:37 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: stop: 2023/07/19 14:38:37 Replica is SECONDARY (2) >Jul 19 14:38:37 ag(ag_cluster)[40121]: INFO: ag_cluster stop : 0 >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: mssql_validate >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: Resource agent invoked with: start >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: mssql_start >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:42 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:42 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:42 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:42 Connected to the instance at localhost:1433 >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Replica is SECONDARY (2) >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Sequence number is 3847:F (61877593833487) >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Line does not match expected syntax. Ignoring. >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61877593833487"] >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61877593833487"] >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 2 sequence numbers were found >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Max sequence number is 3847:F (61877593833487) >Jul 19 14:38:47 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Monitor Caller is: start. >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Replica is SECONDARY (2) >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Instance name is server2. >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: start: 2023/07/19 14:38:47 Current master is . >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:38:48 ag(ag_cluster)[40202]: INFO: ag_cluster start : 0 >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: mssql_validate >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: Resource agent invoked with: notify >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: mssql_notify post-start >Jul 19 14:38:48 ag(ag_cluster)[40402]: INFO: ag_cluster notify : 0 >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: mssql_validate >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: Resource agent invoked with: notify >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: mssql_notify pre-demote >Jul 19 14:38:53 ag(ag_cluster)[40448]: INFO: ag_cluster notify : 0 >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: mssql_validate >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: Resource agent invoked with: notify >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: mssql_notify post-demote >Jul 19 14:38:53 ag(ag_cluster)[40493]: INFO: ag_cluster notify : 0 >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: mssql_validate >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: Resource agent invoked with: notify >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: mssql_notify pre-stop >Jul 19 14:38:53 ag(ag_cluster)[40538]: INFO: ag_cluster notify : 0 >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: mssql_validate >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: Resource agent invoked with: notify >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: mssql_notify post-stop >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: notify: 2023/07/19 14:38:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: notify: 2023/07/19 14:38:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: notify: 2023/07/19 14:38:53 Replica is SECONDARY (2) >Jul 19 14:38:53 ag(ag_cluster)[40583]: INFO: ag_cluster notify : 0 >Jul 19 14:38:53 ag(ag_cluster)[40658]: INFO: mssql_validate >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: Resource agent invoked with: notify >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: mssql_notify pre-promote >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: notify: 2023/07/19 14:38:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: notify: 2023/07/19 14:38:54 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:38:54 ag(ag_cluster)[40658]: INFO: notify: 2023/07/19 14:38:54 Connected to the instance at localhost:1433 >Jul 19 14:38:59 ag(ag_cluster)[40658]: INFO: notify: 2023/07/19 14:38:59 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:38:59 ag(ag_cluster)[40658]: INFO: notify: 2023/07/19 14:38:59 Sequence number is 3847:F (61877593833487) >Jul 19 14:38:59 ag(ag_cluster)[40658]: INFO: notify: SEQUENCE_NUMBER: 61877593833487 >Jul 19 14:39:04 ag(ag_cluster)[40658]: INFO: ag_cluster notify : 0 >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: mssql_validate >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: Resource agent invoked with: notify >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: mssql_notify pre-promote >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: notify: 2023/07/19 14:39:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: notify: 2023/07/19 14:39:04 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:39:04 ag(ag_cluster)[40761]: INFO: notify: 2023/07/19 14:39:04 Connected to the instance at localhost:1433 >Jul 19 14:39:09 ag(ag_cluster)[40761]: INFO: notify: 2023/07/19 14:39:09 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:39:09 ag(ag_cluster)[40761]: INFO: notify: 2023/07/19 14:39:09 Sequence number is 3847:F (61877593833487) >Jul 19 14:39:09 ag(ag_cluster)[40761]: INFO: notify: SEQUENCE_NUMBER: 61877593833487 >Jul 19 14:39:14 ag(ag_cluster)[40761]: INFO: ag_cluster notify : 0 >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: mssql_validate >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: Resource agent invoked with: promote >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: mssql_promote >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:25 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:25 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:39:25 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:25 Connected to the instance at localhost:1433 >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Replica is SECONDARY (2) >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Line does not match expected syntax. Ignoring. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61877593833487"] >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61877593833487"] >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 2 sequence numbers were found >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Max sequence number is 3847:F (61877593833487) >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Sequence number of server2 is 3847:F (61877593833487) >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Promoting replica to PRIMARY role... >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Replica is PRIMARY (1) >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Replica is now PRIMARY >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Monitor Caller is: promote. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Replica is PRIMARY (1) >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Instance name is server2. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Current master is server2. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 DB_FAILOVER is ON. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: 2023/07/19 14:39:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:39:30 ag(ag_cluster)[40885]: INFO: ag_cluster promote : 0 >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: mssql_validate >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: Resource agent invoked with: notify >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: mssql_notify post-promote >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: notify: 2023/07/19 14:39:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: notify: 2023/07/19 14:39:30 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: notify: 2023/07/19 14:39:30 Replica is PRIMARY (1) >Jul 19 14:39:30 ag(ag_cluster)[41161]: INFO: ag_cluster notify : 0 >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: mssql_validate >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: Resource agent invoked with: monitor >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: mssql_monitor >Jul 19 14:39:36 IPaddr2(virtualip)[41244]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:39:36 IPaddr2(virtualip)[41244]: INFO: Bringing device eth0 up >Jul 19 14:39:36 IPaddr2(virtualip)[41244]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:36 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:36 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:39:36 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:36 Connected to the instance at localhost:1433 >Jul 19 14:39:40 IPaddr2(virtualip)[41244]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 Monitor Caller is: monitor. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 Replica is PRIMARY (1) >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 Instance name is server2. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 Current master is server2. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 DB_FAILOVER is ON. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: 2023/07/19 14:39:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: lease_expiry after monitor update: >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:39:41 ag(ag_cluster)[41237]: INFO: ag_cluster monitor : 8 >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: mssql_validate >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: Resource agent invoked with: notify >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: mssql_notify pre-stop >Jul 19 14:39:43 ag(ag_cluster)[41547]: INFO: ag_cluster notify : 0 >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: mssql_validate >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: Resource agent invoked with: notify >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: mssql_notify pre-stop >Jul 19 14:39:49 ag(ag_cluster)[41642]: INFO: ag_cluster notify : 0 >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: mssql_validate >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: Resource agent invoked with: monitor >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: mssql_monitor >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:39:52 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:52 Connected to the instance at localhost:1433 >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 Monitor Caller is: monitor. >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 Replica is PRIMARY (1) >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 Instance name is server2. >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 Current master is server2. >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:39:57 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 DB_FAILOVER is ON. >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: monitor: 2023/07/19 14:39:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: lease_expiry after monitor update: >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:39:58 ag(ag_cluster)[41688]: INFO: ag_cluster monitor : 8 >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: mssql_validate >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: Resource agent invoked with: notify >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: mssql_notify pre-stop >Jul 19 14:39:58 ag(ag_cluster)[41913]: INFO: ag_cluster notify : 0 >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: mssql_validate >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: Resource agent invoked with: notify >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: mssql_notify post-stop >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: notify: 2023/07/19 14:39:58 Replica is PRIMARY (1) >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: notify: 2023/07/19 14:39:58 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: notify: 2023/07/19 14:39:58 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:58 ag(ag_cluster)[41958]: INFO: ag_cluster notify : 0 >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: mssql_validate >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: Resource agent invoked with: notify >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: mssql_notify pre-start >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:39:58 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 Replica is PRIMARY (1) >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:58 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: 2023/07/19 14:39:59 Sequence number is 3848:B (61881888800779) >Jul 19 14:39:59 ag(ag_cluster)[42045]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:40:04 ag(ag_cluster)[42045]: INFO: ag_cluster notify : 0 >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: mssql_validate >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: Resource agent invoked with: notify >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: mssql_notify pre-start >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 Replica is PRIMARY (1) >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: 2023/07/19 14:40:04 Sequence number is 3848:B (61881888800779) >Jul 19 14:40:04 ag(ag_cluster)[42155]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:40:09 ag(ag_cluster)[42155]: INFO: ag_cluster notify : 0 >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: mssql_validate >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: Resource agent invoked with: monitor >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: mssql_monitor >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:40:09 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:09 Connected to the instance at localhost:1433 >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 Monitor Caller is: monitor. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 Replica is PRIMARY (1) >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 Instance name is server2. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 Current master is server2. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 DB_FAILOVER is ON. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: 2023/07/19 14:40:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: lease_expiry after monitor update: >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:40:14 ag(ag_cluster)[42309]: INFO: ag_cluster monitor : 8 >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: mssql_validate >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: Resource agent invoked with: notify >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: mssql_notify post-start >Jul 19 14:40:15 ag(ag_cluster)[42493]: INFO: ag_cluster notify : 0 >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: mssql_validate >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: Resource agent invoked with: notify >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: mssql_notify pre-demote >Jul 19 14:40:15 ag(ag_cluster)[42539]: INFO: ag_cluster notify : 0 >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: mssql_validate >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: Resource agent invoked with: notify >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: mssql_notify post-demote >Jul 19 14:40:15 ag(ag_cluster)[42584]: INFO: ag_cluster notify : 0 >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: mssql_validate >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: Resource agent invoked with: notify >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: mssql_notify pre-stop >Jul 19 14:40:15 ag(ag_cluster)[42629]: INFO: ag_cluster notify : 0 >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: mssql_validate >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: Resource agent invoked with: notify >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: mssql_notify post-stop >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: notify: 2023/07/19 14:40:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: notify: 2023/07/19 14:40:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: notify: 2023/07/19 14:40:15 Replica is PRIMARY (1) >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: notify: 2023/07/19 14:40:15 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: notify: 2023/07/19 14:40:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:40:15 ag(ag_cluster)[42674]: INFO: ag_cluster notify : 0 >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: mssql_validate >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: Resource agent invoked with: monitor >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: mssql_monitor >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:40:26 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:26 Connected to the instance at localhost:1433 >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 Monitor Caller is: monitor. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 Replica is PRIMARY (1) >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 Instance name is server2. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 Current master is server2. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 DB_FAILOVER is ON. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: 2023/07/19 14:40:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: lease_expiry after monitor update: >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:40:31 ag(ag_cluster)[42806]: INFO: ag_cluster monitor : 8 >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: mssql_validate >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: Resource agent invoked with: monitor >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: mssql_monitor >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:42 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:42 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:40:42 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:42 Connected to the instance at localhost:1433 >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 Monitor Caller is: monitor. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 Replica is PRIMARY (1) >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 Instance name is server2. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 Current master is server2. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 DB_FAILOVER is ON. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: 2023/07/19 14:40:47 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: lease_expiry after monitor update: >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:40:47 ag(ag_cluster)[43074]: INFO: ag_cluster monitor : 8 >Jul 19 14:40:58 ag(ag_cluster)[43350]: INFO: mssql_validate >Jul 19 14:40:58 ag(ag_cluster)[43350]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:40:58 ag(ag_cluster)[43350]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:40:58 ag(ag_cluster)[43350]: INFO: Resource agent invoked with: monitor >Jul 19 14:40:58 ag(ag_cluster)[43350]: INFO: mssql_monitor >Jul 19 14:40:59 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:40:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:40:59 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:40:59 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:40:59 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:40:59 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:40:59 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:40:59 Connected to the instance at localhost:1433 >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 Monitor Caller is: monitor. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 Replica is PRIMARY (1) >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 Instance name is server2. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 Current master is server2. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 DB_FAILOVER is ON. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: 2023/07/19 14:41:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: lease_expiry after monitor update: >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:41:04 ag(ag_cluster)[43350]: INFO: ag_cluster monitor : 8 >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: mssql_validate >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: Resource agent invoked with: notify >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: mssql_notify pre-stop >Jul 19 14:41:10 ag(ag_cluster)[43581]: INFO: ag_cluster notify : 0 >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: mssql_validate >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: Resource agent invoked with: notify >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: mssql_notify pre-stop >Jul 19 14:41:13 ag(ag_cluster)[43628]: INFO: ag_cluster notify : 0 >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: mssql_validate >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: Resource agent invoked with: monitor >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: mssql_monitor >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:15 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:41:15 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:15 Connected to the instance at localhost:1433 >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 Monitor Caller is: monitor. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 Replica is PRIMARY (1) >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 Instance name is server2. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 Current master is server2. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 DB_FAILOVER is ON. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: 2023/07/19 14:41:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: lease_expiry after monitor update: >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:41:20 ag(ag_cluster)[43674]: INFO: ag_cluster monitor : 8 >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: mssql_validate >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: Resource agent invoked with: notify >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: mssql_notify post-stop >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: notify: 2023/07/19 14:41:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: notify: 2023/07/19 14:41:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: notify: 2023/07/19 14:41:20 Replica is PRIMARY (1) >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: notify: 2023/07/19 14:41:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:41:20 ag(ag_cluster)[43902]: INFO: notify: 2023/07/19 14:41:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:41:21 ag(ag_cluster)[43902]: INFO: ag_cluster notify : 0 >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: mssql_validate >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: Resource agent invoked with: notify >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: mssql_notify pre-start >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 Replica is PRIMARY (1) >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: 2023/07/19 14:41:21 Sequence number is 3848:B (61881888800779) >Jul 19 14:41:21 ag(ag_cluster)[43989]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:41:26 ag(ag_cluster)[43989]: INFO: ag_cluster notify : 0 >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: mssql_validate >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: Resource agent invoked with: notify >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: mssql_notify pre-start >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 Replica is PRIMARY (1) >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: 2023/07/19 14:41:26 Sequence number is 3848:B (61881888800779) >Jul 19 14:41:26 ag(ag_cluster)[44101]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:41:31 ag(ag_cluster)[44101]: INFO: ag_cluster notify : 0 >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: mssql_validate >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: Resource agent invoked with: monitor >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: mssql_monitor >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:31 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:41:31 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:31 Connected to the instance at localhost:1433 >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:36 Monitor Caller is: monitor. >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Replica is PRIMARY (1) >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Offlining replica... >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Replica is RESOLVING (0) >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Instance name is server2. >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Current master is server1 >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: server2. >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Setting the role to Secondary. >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Setting replica to SECONDARY role... >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: 2023/07/19 14:41:37 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: lease_expiry after monitor update: >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:41:37 ag(ag_cluster)[44254]: INFO: ag_cluster monitor : 0 >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: mssql_validate >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: Resource agent invoked with: notify >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: mssql_notify post-start >Jul 19 14:41:37 ag(ag_cluster)[44473]: INFO: ag_cluster notify : 0 >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: mssql_validate >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: Resource agent invoked with: notify >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: mssql_notify pre-demote >Jul 19 14:41:37 IPaddr2(virtualip)[44538]: INFO: IP status = ok, IP_CIP= >Jul 19 14:41:37 ag(ag_cluster)[44539]: INFO: ag_cluster notify : 0 >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: mssql_validate >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: Resource agent invoked with: demote >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: mssql_demote >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: demote: 2023/07/19 14:41:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: demote: 2023/07/19 14:41:37 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: demote: 2023/07/19 14:41:37 Replica is RESOLVING (0) >Jul 19 14:41:37 ag(ag_cluster)[44637]: INFO: ag_cluster demote : 0 >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: mssql_validate >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: Resource agent invoked with: notify >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: mssql_notify post-demote >Jul 19 14:41:37 ag(ag_cluster)[44730]: INFO: ag_cluster notify : 0 >Jul 19 14:41:37 ag(ag_cluster)[44775]: INFO: mssql_validate >Jul 19 14:41:37 ag(ag_cluster)[44775]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:37 ag(ag_cluster)[44775]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:38 ag(ag_cluster)[44775]: INFO: Resource agent invoked with: notify >Jul 19 14:41:38 ag(ag_cluster)[44775]: INFO: mssql_notify pre-stop >Jul 19 14:41:38 ag(ag_cluster)[44775]: INFO: ag_cluster notify : 0 >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: mssql_validate >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: Resource agent invoked with: stop >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: mssql_stop >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: stop: 2023/07/19 14:41:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: stop: 2023/07/19 14:41:38 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: stop: 2023/07/19 14:41:38 Replica is RESOLVING (0) >Jul 19 14:41:38 ag(ag_cluster)[44820]: INFO: ag_cluster stop : 0 >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: mssql_validate >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: Resource agent invoked with: start >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: mssql_start >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:48 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:41:48 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:48 Connected to the instance at localhost:1433 >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Replica is RESOLVING (0) >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Sequence number is 3848:B (61881888800779) >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Line does not match expected syntax. Ignoring. >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61881888800779"] >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:41:53 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Line does not match expected syntax. Ignoring. >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 1 sequence numbers were found >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Max sequence number is 3848:B (61881888800779) >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Monitor Caller is: start. >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Replica is RESOLVING (0) >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Instance name is server2. >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: 2023/07/19 14:41:53 Current master is . >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:41:54 ag(ag_cluster)[44914]: INFO: ag_cluster start : 0 >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: mssql_validate >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: Resource agent invoked with: notify >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: mssql_notify post-start >Jul 19 14:41:54 ag(ag_cluster)[45112]: INFO: ag_cluster notify : 0 >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: mssql_validate >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: Resource agent invoked with: notify >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: mssql_notify pre-promote >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: notify: 2023/07/19 14:41:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: notify: 2023/07/19 14:41:54 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:41:54 ag(ag_cluster)[45158]: INFO: notify: 2023/07/19 14:41:54 Connected to the instance at localhost:1433 >Jul 19 14:41:59 ag(ag_cluster)[45158]: INFO: notify: 2023/07/19 14:41:59 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:41:59 ag(ag_cluster)[45158]: INFO: notify: 2023/07/19 14:41:59 Sequence number is 3848:B (61881888800779) >Jul 19 14:41:59 ag(ag_cluster)[45158]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:42:04 ag(ag_cluster)[45158]: INFO: ag_cluster notify : 0 >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: mssql_validate >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: Resource agent invoked with: notify >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: mssql_notify pre-promote >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: notify: 2023/07/19 14:42:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: notify: 2023/07/19 14:42:04 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:42:04 ag(ag_cluster)[45266]: INFO: notify: 2023/07/19 14:42:04 Connected to the instance at localhost:1433 >Jul 19 14:42:09 ag(ag_cluster)[45266]: INFO: notify: 2023/07/19 14:42:09 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:42:09 ag(ag_cluster)[45266]: INFO: notify: 2023/07/19 14:42:09 Sequence number is 3848:B (61881888800779) >Jul 19 14:42:09 ag(ag_cluster)[45266]: INFO: notify: SEQUENCE_NUMBER: 61881888800779 >Jul 19 14:42:14 ag(ag_cluster)[45266]: INFO: ag_cluster notify : 0 >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: mssql_validate >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: Resource agent invoked with: promote >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: mssql_promote >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:15 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:15 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:42:15 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:15 Connected to the instance at localhost:1433 >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Replica is RESOLVING (0) >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Line does not match expected syntax. Ignoring. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61881888800779"] >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61881888800779"] >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 2 sequence numbers were found >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Max sequence number is 3848:B (61881888800779) >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Sequence number of server2 is 3848:B (61881888800779) >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Promoting replica to PRIMARY role... >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Replica is PRIMARY (1) >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Replica is now PRIMARY >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Monitor Caller is: promote. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Replica is PRIMARY (1) >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Instance name is server2. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Current master is server2. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 DB_FAILOVER is ON. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: 2023/07/19 14:42:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:42:20 ag(ag_cluster)[45371]: INFO: ag_cluster promote : 0 >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: mssql_validate >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: Resource agent invoked with: notify >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: mssql_notify post-promote >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: notify: 2023/07/19 14:42:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: notify: 2023/07/19 14:42:20 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: notify: 2023/07/19 14:42:20 Replica is PRIMARY (1) >Jul 19 14:42:20 ag(ag_cluster)[45642]: INFO: ag_cluster notify : 0 >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: mssql_validate >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: Resource agent invoked with: monitor >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: mssql_monitor >Jul 19 14:42:20 IPaddr2(virtualip)[45726]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:42:20 IPaddr2(virtualip)[45726]: INFO: Bringing device eth0 up >Jul 19 14:42:20 IPaddr2(virtualip)[45726]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:20 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:42:20 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:20 Connected to the instance at localhost:1433 >Jul 19 14:42:24 IPaddr2(virtualip)[45726]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 Monitor Caller is: monitor. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 Replica is PRIMARY (1) >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 Instance name is server2. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 Current master is server2. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 DB_FAILOVER is ON. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: 2023/07/19 14:42:25 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:25 ag(ag_cluster)[45718]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:42:26 ag(ag_cluster)[45718]: INFO: lease_expiry after monitor update: >Jul 19 14:42:26 ag(ag_cluster)[45718]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:42:26 ag(ag_cluster)[45718]: INFO: ag_cluster monitor : 8 >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: mssql_validate >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: Resource agent invoked with: notify >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: mssql_notify pre-stop >Jul 19 14:42:32 ag(ag_cluster)[46073]: INFO: ag_cluster notify : 0 >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: mssql_validate >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: Resource agent invoked with: notify >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: mssql_notify pre-stop >Jul 19 14:42:32 ag(ag_cluster)[46118]: INFO: ag_cluster notify : 0 >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: mssql_validate >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: Resource agent invoked with: notify >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: mssql_notify post-stop >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: notify: 2023/07/19 14:42:33 Replica is PRIMARY (1) >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: notify: 2023/07/19 14:42:33 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: notify: 2023/07/19 14:42:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:33 ag(ag_cluster)[46165]: INFO: ag_cluster notify : 0 >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: mssql_validate >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: Resource agent invoked with: notify >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: mssql_notify pre-start >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 Replica is PRIMARY (1) >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: 2023/07/19 14:42:33 Sequence number is 3849:6 (61886183768070) >Jul 19 14:42:33 ag(ag_cluster)[46253]: INFO: notify: SEQUENCE_NUMBER: 61886183768070 >Jul 19 14:42:38 ag(ag_cluster)[46253]: INFO: ag_cluster notify : 0 >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: mssql_validate >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: Resource agent invoked with: monitor >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: mssql_monitor >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:42:38 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:38 Connected to the instance at localhost:1433 >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 Monitor Caller is: monitor. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 Replica is PRIMARY (1) >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 Instance name is server2. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 Current master is server2. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 DB_FAILOVER is ON. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: 2023/07/19 14:42:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: lease_expiry after monitor update: >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:42:43 ag(ag_cluster)[46365]: INFO: ag_cluster monitor : 8 >Jul 19 14:42:43 ag(ag_cluster)[46591]: INFO: mssql_validate >Jul 19 14:42:43 ag(ag_cluster)[46591]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:43 ag(ag_cluster)[46591]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: Resource agent invoked with: notify >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: mssql_notify pre-start >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 Replica is PRIMARY (1) >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: 2023/07/19 14:42:44 Sequence number is 3849:B (61886183768075) >Jul 19 14:42:44 ag(ag_cluster)[46591]: INFO: notify: SEQUENCE_NUMBER: 61886183768075 >Jul 19 14:42:49 ag(ag_cluster)[46591]: INFO: ag_cluster notify : 0 >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: mssql_validate >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: Resource agent invoked with: notify >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: mssql_notify pre-start >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 Replica is PRIMARY (1) >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: 2023/07/19 14:42:49 Sequence number is 3849:B (61886183768075) >Jul 19 14:42:49 ag(ag_cluster)[46711]: INFO: notify: SEQUENCE_NUMBER: 61886183768075 >Jul 19 14:42:54 ag(ag_cluster)[46711]: INFO: ag_cluster notify : 0 >Jul 19 14:42:54 ag(ag_cluster)[46866]: INFO: mssql_validate >Jul 19 14:42:54 ag(ag_cluster)[46866]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:42:54 ag(ag_cluster)[46866]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: Resource agent invoked with: monitor >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: mssql_monitor >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:42:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:42:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:42:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:42:55 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:42:55 Connected to the instance at localhost:1433 >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Monitor Caller is: monitor. >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Replica is PRIMARY (1) >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Offlining replica... >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Replica is RESOLVING (0) >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Instance name is server2. >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Current master is server1 >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: server2. >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Setting the role to Secondary. >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Setting replica to SECONDARY role... >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: 2023/07/19 14:43:00 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: lease_expiry after monitor update: >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:43:00 ag(ag_cluster)[46866]: INFO: ag_cluster monitor : 0 >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: mssql_validate >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: Resource agent invoked with: notify >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: mssql_notify post-start >Jul 19 14:43:00 ag(ag_cluster)[47063]: INFO: ag_cluster notify : 0 >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: mssql_validate >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: Resource agent invoked with: notify >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: mssql_notify pre-demote >Jul 19 14:43:00 IPaddr2(virtualip)[47109]: INFO: IP status = ok, IP_CIP= >Jul 19 14:43:00 ag(ag_cluster)[47110]: INFO: ag_cluster notify : 0 >Jul 19 14:43:00 ag(ag_cluster)[47208]: INFO: mssql_validate >Jul 19 14:43:00 ag(ag_cluster)[47208]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:00 ag(ag_cluster)[47208]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:43:00 ag(ag_cluster)[47208]: INFO: Resource agent invoked with: demote >Jul 19 14:43:01 ag(ag_cluster)[47208]: INFO: mssql_demote >Jul 19 14:43:01 ag(ag_cluster)[47208]: INFO: demote: 2023/07/19 14:43:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:43:01 ag(ag_cluster)[47208]: INFO: demote: 2023/07/19 14:43:01 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:01 ag(ag_cluster)[47208]: INFO: demote: 2023/07/19 14:43:01 Replica is RESOLVING (0) >Jul 19 14:43:01 ag(ag_cluster)[47208]: INFO: ag_cluster demote : 0 >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: mssql_validate >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: Resource agent invoked with: notify >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: mssql_notify post-demote >Jul 19 14:43:01 ag(ag_cluster)[47301]: INFO: ag_cluster notify : 0 >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: mssql_validate >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: Resource agent invoked with: notify >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: mssql_notify pre-stop >Jul 19 14:43:01 ag(ag_cluster)[47346]: INFO: ag_cluster notify : 0 >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: mssql_validate >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: Resource agent invoked with: stop >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: mssql_stop >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: stop: 2023/07/19 14:43:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: stop: 2023/07/19 14:43:01 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: stop: 2023/07/19 14:43:01 Replica is RESOLVING (0) >Jul 19 14:43:01 ag(ag_cluster)[47391]: INFO: ag_cluster stop : 0 >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: mssql_validate >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: Resource agent invoked with: start >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: mssql_start >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:12 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:12 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:43:12 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:12 Connected to the instance at localhost:1433 >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Replica is RESOLVING (0) >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Sequence number is 3849:B (61886183768075) >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Line does not match expected syntax. Ignoring. >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61886183768075"] >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Line does not match expected syntax. Ignoring. >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 1 sequence numbers were found >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Max sequence number is 3849:B (61886183768075) >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Monitor Caller is: start. >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Replica is RESOLVING (0) >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Instance name is server2. >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: 2023/07/19 14:43:17 Current master is . >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:43:17 ag(ag_cluster)[47474]: INFO: ag_cluster start : 0 >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: mssql_validate >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: Resource agent invoked with: notify >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: mssql_notify post-start >Jul 19 14:43:17 ag(ag_cluster)[47671]: INFO: ag_cluster notify : 0 >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: mssql_validate >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: Resource agent invoked with: notify >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: mssql_notify pre-promote >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: notify: 2023/07/19 14:43:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: notify: 2023/07/19 14:43:17 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:43:17 ag(ag_cluster)[47717]: INFO: notify: 2023/07/19 14:43:17 Connected to the instance at localhost:1433 >Jul 19 14:43:22 ag(ag_cluster)[47717]: INFO: notify: 2023/07/19 14:43:22 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:43:22 ag(ag_cluster)[47717]: INFO: notify: 2023/07/19 14:43:22 Sequence number is 3849:B (61886183768075) >Jul 19 14:43:22 ag(ag_cluster)[47717]: INFO: notify: SEQUENCE_NUMBER: 61886183768075 >Jul 19 14:43:27 ag(ag_cluster)[47717]: INFO: ag_cluster notify : 0 >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: mssql_validate >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: Resource agent invoked with: notify >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: mssql_notify pre-promote >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: notify: 2023/07/19 14:43:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: notify: 2023/07/19 14:43:28 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:43:28 ag(ag_cluster)[47823]: INFO: notify: 2023/07/19 14:43:28 Connected to the instance at localhost:1433 >Jul 19 14:43:33 ag(ag_cluster)[47823]: INFO: notify: 2023/07/19 14:43:33 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:43:33 ag(ag_cluster)[47823]: INFO: notify: 2023/07/19 14:43:33 Sequence number is 3849:B (61886183768075) >Jul 19 14:43:33 ag(ag_cluster)[47823]: INFO: notify: SEQUENCE_NUMBER: 61886183768075 >Jul 19 14:43:38 ag(ag_cluster)[47823]: INFO: ag_cluster notify : 0 >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: mssql_validate >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: Resource agent invoked with: promote >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: mssql_promote >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:38 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:38 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:43:38 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:38 Connected to the instance at localhost:1433 >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Replica is RESOLVING (0) >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Line does not match expected syntax. Ignoring. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61886183768075"] >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61886183768075"] >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 2 sequence numbers were found >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Max sequence number is 3849:B (61886183768075) >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Sequence number of server2 is 3849:B (61886183768075) >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Promoting replica to PRIMARY role... >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Replica is PRIMARY (1) >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Replica is now PRIMARY >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Monitor Caller is: promote. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Replica is PRIMARY (1) >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Instance name is server2. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Current master is server2. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 DB_FAILOVER is ON. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: 2023/07/19 14:43:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:43:43 ag(ag_cluster)[47930]: INFO: ag_cluster promote : 0 >Jul 19 14:43:43 ag(ag_cluster)[48204]: INFO: mssql_validate >Jul 19 14:43:43 ag(ag_cluster)[48204]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:43 ag(ag_cluster)[48204]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:43 ag(ag_cluster)[48204]: INFO: Resource agent invoked with: notify >Jul 19 14:43:43 ag(ag_cluster)[48204]: INFO: mssql_notify post-promote >Jul 19 14:43:44 ag(ag_cluster)[48204]: INFO: notify: 2023/07/19 14:43:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:43:44 ag(ag_cluster)[48204]: INFO: notify: 2023/07/19 14:43:44 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:44 ag(ag_cluster)[48204]: INFO: notify: 2023/07/19 14:43:44 Replica is PRIMARY (1) >Jul 19 14:43:44 ag(ag_cluster)[48204]: INFO: ag_cluster notify : 0 >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: mssql_validate >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: Resource agent invoked with: monitor >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: mssql_monitor >Jul 19 14:43:44 IPaddr2(virtualip)[48292]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:43:44 IPaddr2(virtualip)[48292]: INFO: Bringing device eth0 up >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:43:44 IPaddr2(virtualip)[48292]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:44 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:44 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:43:44 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:44 Connected to the instance at localhost:1433 >Jul 19 14:43:48 IPaddr2(virtualip)[48292]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 Monitor Caller is: monitor. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 Replica is PRIMARY (1) >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 Instance name is server2. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 Current master is server2. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 DB_FAILOVER is ON. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: 2023/07/19 14:43:49 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: lease_expiry after monitor update: >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:43:49 ag(ag_cluster)[48280]: INFO: ag_cluster monitor : 8 >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: mssql_validate >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: Resource agent invoked with: notify >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: mssql_notify pre-stop >Jul 19 14:43:55 ag(ag_cluster)[48646]: INFO: ag_cluster notify : 0 >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: mssql_validate >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: Resource agent invoked with: notify >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: mssql_notify pre-stop >Jul 19 14:43:55 ag(ag_cluster)[48691]: INFO: ag_cluster notify : 0 >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: mssql_validate >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: Resource agent invoked with: notify >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: mssql_notify post-stop >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: notify: 2023/07/19 14:43:56 Replica is PRIMARY (1) >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: notify: 2023/07/19 14:43:56 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: notify: 2023/07/19 14:43:56 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:43:56 ag(ag_cluster)[48736]: INFO: ag_cluster notify : 0 >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: mssql_validate >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: Resource agent invoked with: notify >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: mssql_notify pre-start >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 Replica is PRIMARY (1) >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: 2023/07/19 14:43:56 Sequence number is 384A:6 (61890478735366) >Jul 19 14:43:56 ag(ag_cluster)[48822]: INFO: notify: SEQUENCE_NUMBER: 61890478735366 >Jul 19 14:44:01 ag(ag_cluster)[48822]: INFO: ag_cluster notify : 0 >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: mssql_validate >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: Resource agent invoked with: monitor >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: mssql_monitor >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:01 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:01 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:44:01 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:01 Connected to the instance at localhost:1433 >Jul 19 14:44:06 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 Monitor Caller is: monitor. >Jul 19 14:44:06 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:06 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 Replica is PRIMARY (1) >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 Instance name is server2. >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 Current master is server2. >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 DB_FAILOVER is ON. >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: 2023/07/19 14:44:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: lease_expiry after monitor update: >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:44:07 ag(ag_cluster)[48936]: INFO: ag_cluster monitor : 8 >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: mssql_validate >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: Resource agent invoked with: notify >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: mssql_notify pre-start >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 Replica is PRIMARY (1) >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: 2023/07/19 14:44:07 Sequence number is 384A:B (61890478735371) >Jul 19 14:44:07 ag(ag_cluster)[49164]: INFO: notify: SEQUENCE_NUMBER: 61890478735371 >Jul 19 14:44:12 ag(ag_cluster)[49164]: INFO: ag_cluster notify : 0 >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: mssql_validate >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: Resource agent invoked with: notify >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: mssql_notify pre-start >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 Replica is PRIMARY (1) >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: 2023/07/19 14:44:12 Sequence number is 384A:B (61890478735371) >Jul 19 14:44:12 ag(ag_cluster)[49277]: INFO: notify: SEQUENCE_NUMBER: 61890478735371 >Jul 19 14:44:17 ag(ag_cluster)[49277]: INFO: ag_cluster notify : 0 >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: mssql_validate >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: Resource agent invoked with: monitor >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: mssql_monitor >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:44:18 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:18 Connected to the instance at localhost:1433 >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Monitor Caller is: monitor. >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Replica is PRIMARY (1) >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Offlining replica... >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Replica is RESOLVING (0) >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Instance name is server2. >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Current master is server1 >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: server2. >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Setting the role to Secondary. >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Setting replica to SECONDARY role... >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: 2023/07/19 14:44:23 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: lease_expiry after monitor update: >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:44:23 ag(ag_cluster)[49432]: INFO: ag_cluster monitor : 0 >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: mssql_validate >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: Resource agent invoked with: notify >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: mssql_notify post-start >Jul 19 14:44:23 ag(ag_cluster)[49635]: INFO: ag_cluster notify : 0 >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: mssql_validate >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: Resource agent invoked with: notify >Jul 19 14:44:23 IPaddr2(virtualip)[49681]: INFO: IP status = ok, IP_CIP= >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: mssql_notify pre-demote >Jul 19 14:44:23 ag(ag_cluster)[49682]: INFO: ag_cluster notify : 0 >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: mssql_validate >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: Resource agent invoked with: demote >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: mssql_demote >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: demote: 2023/07/19 14:44:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: demote: 2023/07/19 14:44:23 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: demote: 2023/07/19 14:44:23 Replica is RESOLVING (0) >Jul 19 14:44:23 ag(ag_cluster)[49780]: INFO: ag_cluster demote : 0 >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: mssql_validate >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: Resource agent invoked with: notify >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: mssql_notify post-demote >Jul 19 14:44:23 ag(ag_cluster)[49873]: INFO: ag_cluster notify : 0 >Jul 19 14:44:23 ag(ag_cluster)[49918]: INFO: mssql_validate >Jul 19 14:44:23 ag(ag_cluster)[49918]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:23 ag(ag_cluster)[49918]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:24 ag(ag_cluster)[49918]: INFO: Resource agent invoked with: notify >Jul 19 14:44:24 ag(ag_cluster)[49918]: INFO: mssql_notify pre-stop >Jul 19 14:44:24 ag(ag_cluster)[49918]: INFO: ag_cluster notify : 0 >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: mssql_validate >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: Resource agent invoked with: stop >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: mssql_stop >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: stop: 2023/07/19 14:44:24 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: stop: 2023/07/19 14:44:24 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: stop: 2023/07/19 14:44:24 Replica is RESOLVING (0) >Jul 19 14:44:24 ag(ag_cluster)[49963]: INFO: ag_cluster stop : 0 >Jul 19 14:44:34 ag(ag_cluster)[50047]: INFO: mssql_validate >Jul 19 14:44:34 ag(ag_cluster)[50047]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:34 ag(ag_cluster)[50047]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:34 ag(ag_cluster)[50047]: INFO: Resource agent invoked with: start >Jul 19 14:44:34 ag(ag_cluster)[50047]: INFO: mssql_start >Jul 19 14:44:35 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:44:35 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:35 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:44:35 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:35 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:44:35 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:35 Connected to the instance at localhost:1433 >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Replica is RESOLVING (0) >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Sequence number is 384A:B (61890478735371) >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Line does not match expected syntax. Ignoring. >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61890478735371"] >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Line does not match expected syntax. Ignoring. >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 1 sequence numbers were found >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Max sequence number is 384A:B (61890478735371) >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Monitor Caller is: start. >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Replica is RESOLVING (0) >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Instance name is server2. >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: 2023/07/19 14:44:40 Current master is . >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:44:40 ag(ag_cluster)[50047]: INFO: ag_cluster start : 0 >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: mssql_validate >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: Resource agent invoked with: notify >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: mssql_notify post-start >Jul 19 14:44:40 ag(ag_cluster)[50249]: INFO: ag_cluster notify : 0 >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: mssql_validate >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: Resource agent invoked with: notify >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: mssql_notify pre-promote >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: notify: 2023/07/19 14:44:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: notify: 2023/07/19 14:44:40 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:44:40 ag(ag_cluster)[50295]: INFO: notify: 2023/07/19 14:44:40 Connected to the instance at localhost:1433 >Jul 19 14:44:45 ag(ag_cluster)[50295]: INFO: notify: 2023/07/19 14:44:45 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:44:45 ag(ag_cluster)[50295]: INFO: notify: 2023/07/19 14:44:45 Sequence number is 384A:B (61890478735371) >Jul 19 14:44:45 ag(ag_cluster)[50295]: INFO: notify: SEQUENCE_NUMBER: 61890478735371 >Jul 19 14:44:50 ag(ag_cluster)[50295]: INFO: ag_cluster notify : 0 >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: mssql_validate >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: Resource agent invoked with: notify >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: mssql_notify pre-promote >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: notify: 2023/07/19 14:44:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: notify: 2023/07/19 14:44:51 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:44:51 ag(ag_cluster)[50409]: INFO: notify: 2023/07/19 14:44:51 Connected to the instance at localhost:1433 >Jul 19 14:44:56 ag(ag_cluster)[50409]: INFO: notify: 2023/07/19 14:44:56 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:44:56 ag(ag_cluster)[50409]: INFO: notify: 2023/07/19 14:44:56 Sequence number is 384A:B (61890478735371) >Jul 19 14:44:56 ag(ag_cluster)[50409]: INFO: notify: SEQUENCE_NUMBER: 61890478735371 >Jul 19 14:45:01 ag(ag_cluster)[50409]: INFO: ag_cluster notify : 0 >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: mssql_validate >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: Resource agent invoked with: promote >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: mssql_promote >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:01 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:01 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:45:01 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:01 Connected to the instance at localhost:1433 >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Replica is RESOLVING (0) >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Line does not match expected syntax. Ignoring. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61890478735371"] >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61890478735371"] >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 2 sequence numbers were found >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Max sequence number is 384A:B (61890478735371) >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Sequence number of server2 is 384A:B (61890478735371) >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Promoting replica to PRIMARY role... >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Replica is PRIMARY (1) >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Replica is now PRIMARY >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Monitor Caller is: promote. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Replica is PRIMARY (1) >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Instance name is server2. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Current master is server2. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 DB_FAILOVER is ON. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: 2023/07/19 14:45:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:45:06 ag(ag_cluster)[50514]: INFO: ag_cluster promote : 0 >Jul 19 14:45:06 ag(ag_cluster)[50788]: INFO: mssql_validate >Jul 19 14:45:06 ag(ag_cluster)[50788]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:06 ag(ag_cluster)[50788]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: Resource agent invoked with: notify >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: mssql_notify post-promote >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: notify: 2023/07/19 14:45:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: notify: 2023/07/19 14:45:07 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: notify: 2023/07/19 14:45:07 Replica is PRIMARY (1) >Jul 19 14:45:07 ag(ag_cluster)[50788]: INFO: ag_cluster notify : 0 >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: mssql_validate >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: Resource agent invoked with: monitor >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: mssql_monitor >Jul 19 14:45:07 IPaddr2(virtualip)[50878]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:45:07 IPaddr2(virtualip)[50878]: INFO: Bringing device eth0 up >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:45:07 IPaddr2(virtualip)[50878]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:07 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:45:07 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:07 Connected to the instance at localhost:1433 >Jul 19 14:45:11 IPaddr2(virtualip)[50878]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 Monitor Caller is: monitor. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 Replica is PRIMARY (1) >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 Instance name is server2. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 Current master is server2. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 DB_FAILOVER is ON. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: 2023/07/19 14:45:12 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: lease_expiry after monitor update: >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:45:12 ag(ag_cluster)[50864]: INFO: ag_cluster monitor : 8 >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: mssql_validate >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: Resource agent invoked with: notify >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: mssql_notify pre-stop >Jul 19 14:45:20 ag(ag_cluster)[51220]: INFO: ag_cluster notify : 0 >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: mssql_validate >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: Resource agent invoked with: notify >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: mssql_notify pre-stop >Jul 19 14:45:20 ag(ag_cluster)[51265]: INFO: ag_cluster notify : 0 >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: mssql_validate >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: Resource agent invoked with: notify >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: mssql_notify post-stop >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: notify: 2023/07/19 14:45:20 Replica is PRIMARY (1) >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: notify: 2023/07/19 14:45:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: notify: 2023/07/19 14:45:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:20 ag(ag_cluster)[51310]: INFO: ag_cluster notify : 0 >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: mssql_validate >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: Resource agent invoked with: notify >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: mssql_notify pre-start >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 Replica is PRIMARY (1) >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: 2023/07/19 14:45:20 Sequence number is 384B:8 (61894773702664) >Jul 19 14:45:20 ag(ag_cluster)[51397]: INFO: notify: SEQUENCE_NUMBER: 61894773702664 >Jul 19 14:45:25 ag(ag_cluster)[51397]: INFO: ag_cluster notify : 0 >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: mssql_validate >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: Resource agent invoked with: monitor >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: mssql_monitor >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:45:26 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:26 Connected to the instance at localhost:1433 >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 Monitor Caller is: monitor. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 Replica is PRIMARY (1) >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 Instance name is server2. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 Current master is server2. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 DB_FAILOVER is ON. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: 2023/07/19 14:45:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: lease_expiry after monitor update: >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:45:31 ag(ag_cluster)[51511]: INFO: ag_cluster monitor : 8 >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: mssql_validate >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: Resource agent invoked with: notify >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: mssql_notify pre-start >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 Replica is PRIMARY (1) >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: 2023/07/19 14:45:31 Sequence number is 384B:B (61894773702667) >Jul 19 14:45:31 ag(ag_cluster)[51737]: INFO: notify: SEQUENCE_NUMBER: 61894773702667 >Jul 19 14:45:36 ag(ag_cluster)[51737]: INFO: ag_cluster notify : 0 >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: mssql_validate >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: Resource agent invoked with: notify >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: mssql_notify pre-start >Jul 19 14:45:36 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:36 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:37 Replica is PRIMARY (1) >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:37 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:37 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:37 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: 2023/07/19 14:45:37 Sequence number is 384B:B (61894773702667) >Jul 19 14:45:37 ag(ag_cluster)[51849]: INFO: notify: SEQUENCE_NUMBER: 61894773702667 >Jul 19 14:45:42 ag(ag_cluster)[51849]: INFO: ag_cluster notify : 0 >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: mssql_validate >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: Resource agent invoked with: monitor >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: mssql_monitor >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:42 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:42 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:45:42 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:42 Connected to the instance at localhost:1433 >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Monitor Caller is: monitor. >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Replica is PRIMARY (1) >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Offlining replica... >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Replica is RESOLVING (0) >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Instance name is server2. >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Current master is server1 >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: server2. >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Setting the role to Secondary. >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Setting replica to SECONDARY role... >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: 2023/07/19 14:45:47 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: lease_expiry after monitor update: >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:45:47 ag(ag_cluster)[52004]: INFO: ag_cluster monitor : 0 >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: Resource agent invoked with: notify >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: mssql_notify post-start >Jul 19 14:45:48 ag(ag_cluster)[52252]: INFO: ag_cluster notify : 0 >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: Resource agent invoked with: notify >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: mssql_notify pre-demote >Jul 19 14:45:48 IPaddr2(virtualip)[52298]: INFO: IP status = ok, IP_CIP= >Jul 19 14:45:48 ag(ag_cluster)[52299]: INFO: ag_cluster notify : 0 >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: Resource agent invoked with: demote >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: mssql_demote >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: demote: 2023/07/19 14:45:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: demote: 2023/07/19 14:45:48 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: demote: 2023/07/19 14:45:48 Replica is RESOLVING (0) >Jul 19 14:45:48 ag(ag_cluster)[52397]: INFO: ag_cluster demote : 0 >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: Resource agent invoked with: notify >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: mssql_notify post-demote >Jul 19 14:45:48 ag(ag_cluster)[52490]: INFO: ag_cluster notify : 0 >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: Resource agent invoked with: notify >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: mssql_notify pre-stop >Jul 19 14:45:48 ag(ag_cluster)[52535]: INFO: ag_cluster notify : 0 >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: mssql_validate >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: Resource agent invoked with: stop >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: mssql_stop >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: stop: 2023/07/19 14:45:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: stop: 2023/07/19 14:45:48 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: stop: 2023/07/19 14:45:48 Replica is RESOLVING (0) >Jul 19 14:45:48 ag(ag_cluster)[52580]: INFO: ag_cluster stop : 0 >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: mssql_validate >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: Resource agent invoked with: start >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: mssql_start >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:45:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:45:59 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:45:59 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:45:59 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:45:59 Connected to the instance at localhost:1433 >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Replica is RESOLVING (0) >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Sequence number is 384B:B (61894773702667) >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Line does not match expected syntax. Ignoring. >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61894773702667"] >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Line does not match expected syntax. Ignoring. >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 1 sequence numbers were found >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Max sequence number is 384B:B (61894773702667) >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Monitor Caller is: start. >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Replica is RESOLVING (0) >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Instance name is server2. >Jul 19 14:46:04 ag(ag_cluster)[52665]: INFO: start: 2023/07/19 14:46:04 Current master is . >Jul 19 14:46:05 ag(ag_cluster)[52665]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:46:05 ag(ag_cluster)[52665]: INFO: ag_cluster start : 0 >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: mssql_validate >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: Resource agent invoked with: notify >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: mssql_notify post-start >Jul 19 14:46:05 ag(ag_cluster)[52865]: INFO: ag_cluster notify : 0 >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: mssql_validate >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: Resource agent invoked with: notify >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: mssql_notify pre-promote >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: notify: 2023/07/19 14:46:05 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: notify: 2023/07/19 14:46:05 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:46:05 ag(ag_cluster)[52911]: INFO: notify: 2023/07/19 14:46:05 Connected to the instance at localhost:1433 >Jul 19 14:46:10 ag(ag_cluster)[52911]: INFO: notify: 2023/07/19 14:46:10 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:46:10 ag(ag_cluster)[52911]: INFO: notify: 2023/07/19 14:46:10 Sequence number is 384B:B (61894773702667) >Jul 19 14:46:10 ag(ag_cluster)[52911]: INFO: notify: SEQUENCE_NUMBER: 61894773702667 >Jul 19 14:46:15 ag(ag_cluster)[52911]: INFO: ag_cluster notify : 0 >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: mssql_validate >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: Resource agent invoked with: notify >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: mssql_notify pre-promote >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: notify: 2023/07/19 14:46:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: notify: 2023/07/19 14:46:15 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:46:15 ag(ag_cluster)[53018]: INFO: notify: 2023/07/19 14:46:15 Connected to the instance at localhost:1433 >Jul 19 14:46:20 ag(ag_cluster)[53018]: INFO: notify: 2023/07/19 14:46:20 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:46:20 ag(ag_cluster)[53018]: INFO: notify: 2023/07/19 14:46:20 Sequence number is 384B:B (61894773702667) >Jul 19 14:46:20 ag(ag_cluster)[53018]: INFO: notify: SEQUENCE_NUMBER: 61894773702667 >Jul 19 14:46:25 ag(ag_cluster)[53018]: INFO: ag_cluster notify : 0 >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: mssql_validate >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: Resource agent invoked with: promote >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: mssql_promote >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:25 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:25 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:46:25 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:25 Connected to the instance at localhost:1433 >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Replica is RESOLVING (0) >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Line does not match expected syntax. Ignoring. >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61894773702667"] >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61894773702667"] >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 2 sequence numbers were found >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Max sequence number is 384B:B (61894773702667) >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Sequence number of server2 is 384B:B (61894773702667) >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Promoting replica to PRIMARY role... >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Replica is PRIMARY (1) >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Replica is now PRIMARY >Jul 19 14:46:30 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Monitor Caller is: promote. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Replica is PRIMARY (1) >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Instance name is server2. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:30 Current master is server2. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:31 DB_FAILOVER is ON. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:31 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: 2023/07/19 14:46:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:46:31 ag(ag_cluster)[53123]: INFO: ag_cluster promote : 0 >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: mssql_validate >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: Resource agent invoked with: notify >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: mssql_notify post-promote >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: notify: 2023/07/19 14:46:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: notify: 2023/07/19 14:46:31 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: notify: 2023/07/19 14:46:31 Replica is PRIMARY (1) >Jul 19 14:46:31 ag(ag_cluster)[53400]: INFO: ag_cluster notify : 0 >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: mssql_validate >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: Resource agent invoked with: monitor >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: mssql_monitor >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:46:31 IPaddr2(virtualip)[53487]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:46:31 IPaddr2(virtualip)[53487]: INFO: Bringing device eth0 up >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:31 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:46:31 IPaddr2(virtualip)[53487]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:46:31 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:31 Connected to the instance at localhost:1433 >Jul 19 14:46:35 IPaddr2(virtualip)[53487]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 Monitor Caller is: monitor. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 Replica is PRIMARY (1) >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 Instance name is server2. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 Current master is server2. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 DB_FAILOVER is ON. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: 2023/07/19 14:46:36 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: lease_expiry after monitor update: >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:46:36 ag(ag_cluster)[53476]: INFO: ag_cluster monitor : 8 >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: mssql_validate >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: Resource agent invoked with: notify >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: mssql_notify pre-stop >Jul 19 14:46:43 ag(ag_cluster)[53831]: INFO: ag_cluster notify : 0 >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: mssql_validate >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: Resource agent invoked with: notify >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: mssql_notify pre-stop >Jul 19 14:46:43 ag(ag_cluster)[53876]: INFO: ag_cluster notify : 0 >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: mssql_validate >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: Resource agent invoked with: notify >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: mssql_notify post-stop >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: notify: 2023/07/19 14:46:43 Replica is PRIMARY (1) >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: notify: 2023/07/19 14:46:43 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: notify: 2023/07/19 14:46:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:43 ag(ag_cluster)[53921]: INFO: ag_cluster notify : 0 >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: mssql_validate >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: Resource agent invoked with: notify >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: mssql_notify pre-start >Jul 19 14:46:43 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:44 Replica is PRIMARY (1) >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:44 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:44 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:44 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: 2023/07/19 14:46:44 Sequence number is 384C:6 (61899068669958) >Jul 19 14:46:44 ag(ag_cluster)[54008]: INFO: notify: SEQUENCE_NUMBER: 61899068669958 >Jul 19 14:46:49 ag(ag_cluster)[54008]: INFO: ag_cluster notify : 0 >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: mssql_validate >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: Resource agent invoked with: monitor >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: mssql_monitor >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:49 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:46:49 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:49 Connected to the instance at localhost:1433 >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 Monitor Caller is: monitor. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 Replica is PRIMARY (1) >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 Instance name is server2. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 Current master is server2. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 DB_FAILOVER is ON. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: 2023/07/19 14:46:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: lease_expiry after monitor update: >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:46:54 ag(ag_cluster)[54128]: INFO: ag_cluster monitor : 8 >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: mssql_validate >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: Resource agent invoked with: notify >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: mssql_notify pre-start >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 Replica is PRIMARY (1) >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: 2023/07/19 14:46:54 Sequence number is 384C:7 (61899068669959) >Jul 19 14:46:54 ag(ag_cluster)[54356]: INFO: notify: SEQUENCE_NUMBER: 61899068669959 >Jul 19 14:47:00 ag(ag_cluster)[54356]: INFO: ag_cluster notify : 0 >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: mssql_validate >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: Resource agent invoked with: notify >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: mssql_notify pre-start >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 Replica is PRIMARY (1) >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: 2023/07/19 14:47:00 Sequence number is 384C:7 (61899068669959) >Jul 19 14:47:00 ag(ag_cluster)[54469]: INFO: notify: SEQUENCE_NUMBER: 61899068669959 >Jul 19 14:47:05 ag(ag_cluster)[54469]: INFO: ag_cluster notify : 0 >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: mssql_validate >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: Resource agent invoked with: monitor >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: mssql_monitor >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:05 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:05 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:05 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:05 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:05 Connected to the instance at localhost:1433 >Jul 19 14:47:10 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:10 Monitor Caller is: monitor. >Jul 19 14:47:10 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:10 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:10 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:10 Replica is PRIMARY (1) >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:10 Offlining replica... >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Replica is RESOLVING (0) >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Instance name is server2. >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Current master is server1 >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: server2. >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Setting the role to Secondary. >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Setting replica to SECONDARY role... >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: 2023/07/19 14:47:11 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: lease_expiry after monitor update: >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:47:11 ag(ag_cluster)[54624]: INFO: ag_cluster monitor : 0 >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: Resource agent invoked with: notify >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: mssql_notify post-start >Jul 19 14:47:11 ag(ag_cluster)[54822]: INFO: ag_cluster notify : 0 >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: Resource agent invoked with: notify >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: mssql_notify pre-demote >Jul 19 14:47:11 IPaddr2(virtualip)[54868]: INFO: IP status = ok, IP_CIP= >Jul 19 14:47:11 ag(ag_cluster)[54869]: INFO: ag_cluster notify : 0 >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: Resource agent invoked with: demote >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: mssql_demote >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: demote: 2023/07/19 14:47:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: demote: 2023/07/19 14:47:11 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: demote: 2023/07/19 14:47:11 Replica is RESOLVING (0) >Jul 19 14:47:11 ag(ag_cluster)[54968]: INFO: ag_cluster demote : 0 >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: Resource agent invoked with: notify >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: mssql_notify post-demote >Jul 19 14:47:11 ag(ag_cluster)[55061]: INFO: ag_cluster notify : 0 >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: Resource agent invoked with: notify >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: mssql_notify pre-stop >Jul 19 14:47:11 ag(ag_cluster)[55106]: INFO: ag_cluster notify : 0 >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: mssql_validate >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: Resource agent invoked with: stop >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: mssql_stop >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: stop: 2023/07/19 14:47:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: stop: 2023/07/19 14:47:11 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: stop: 2023/07/19 14:47:11 Replica is RESOLVING (0) >Jul 19 14:47:11 ag(ag_cluster)[55151]: INFO: ag_cluster stop : 0 >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: mssql_validate >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: Resource agent invoked with: start >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: mssql_start >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:22 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:22 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:22 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:22 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:22 Connected to the instance at localhost:1433 >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Replica is RESOLVING (0) >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Sequence number is 384C:7 (61899068669959) >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Line does not match expected syntax. Ignoring. >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61899068669959"] >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Line does not match expected syntax. Ignoring. >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 1 sequence numbers were found >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Max sequence number is 384C:7 (61899068669959) >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 Monitor Caller is: start. >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:27 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:28 Replica is RESOLVING (0) >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:28 Instance name is server2. >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: 2023/07/19 14:47:28 Current master is . >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:47:28 ag(ag_cluster)[55236]: INFO: ag_cluster start : 0 >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: mssql_validate >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: Resource agent invoked with: notify >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: mssql_notify post-start >Jul 19 14:47:28 ag(ag_cluster)[55435]: INFO: ag_cluster notify : 0 >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: mssql_validate >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: Resource agent invoked with: notify >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: mssql_notify pre-promote >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: notify: 2023/07/19 14:47:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: notify: 2023/07/19 14:47:28 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:28 ag(ag_cluster)[55481]: INFO: notify: 2023/07/19 14:47:28 Connected to the instance at localhost:1433 >Jul 19 14:47:33 ag(ag_cluster)[55481]: INFO: notify: 2023/07/19 14:47:33 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:47:33 ag(ag_cluster)[55481]: INFO: notify: 2023/07/19 14:47:33 Sequence number is 384C:7 (61899068669959) >Jul 19 14:47:33 ag(ag_cluster)[55481]: INFO: notify: SEQUENCE_NUMBER: 61899068669959 >Jul 19 14:47:38 ag(ag_cluster)[55481]: INFO: ag_cluster notify : 0 >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: mssql_validate >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: Resource agent invoked with: notify >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: mssql_notify pre-promote >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: notify: 2023/07/19 14:47:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: notify: 2023/07/19 14:47:38 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:38 ag(ag_cluster)[55587]: INFO: notify: 2023/07/19 14:47:38 Connected to the instance at localhost:1433 >Jul 19 14:47:43 ag(ag_cluster)[55587]: INFO: notify: 2023/07/19 14:47:43 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:47:43 ag(ag_cluster)[55587]: INFO: notify: 2023/07/19 14:47:43 Sequence number is 384C:7 (61899068669959) >Jul 19 14:47:43 ag(ag_cluster)[55587]: INFO: notify: SEQUENCE_NUMBER: 61899068669959 >Jul 19 14:47:48 ag(ag_cluster)[55587]: INFO: ag_cluster notify : 0 >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: mssql_validate >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: Resource agent invoked with: promote >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: mssql_promote >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:49 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:49 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:49 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:49 Connected to the instance at localhost:1433 >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Replica is RESOLVING (0) >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Line does not match expected syntax. Ignoring. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61899068669959"] >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61899068669959"] >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 2 sequence numbers were found >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Max sequence number is 384C:7 (61899068669959) >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Sequence number of server2 is 384C:7 (61899068669959) >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Promoting replica to PRIMARY role... >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Replica is PRIMARY (1) >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Replica is now PRIMARY >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Monitor Caller is: promote. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Replica is PRIMARY (1) >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Instance name is server2. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Current master is server2. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 DB_FAILOVER is ON. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: 2023/07/19 14:47:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:47:54 ag(ag_cluster)[55707]: INFO: ag_cluster promote : 0 >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: mssql_validate >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: Resource agent invoked with: notify >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: mssql_notify post-promote >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: notify: 2023/07/19 14:47:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: notify: 2023/07/19 14:47:54 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: notify: 2023/07/19 14:47:54 Replica is PRIMARY (1) >Jul 19 14:47:54 ag(ag_cluster)[55980]: INFO: ag_cluster notify : 0 >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: mssql_validate >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: Resource agent invoked with: monitor >Jul 19 14:47:54 IPaddr2(virtualip)[56063]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: mssql_monitor >Jul 19 14:47:54 IPaddr2(virtualip)[56063]: INFO: Bringing device eth0 up >Jul 19 14:47:54 IPaddr2(virtualip)[56063]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:54 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:54 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:47:54 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:54 Connected to the instance at localhost:1433 >Jul 19 14:47:58 IPaddr2(virtualip)[56063]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 Monitor Caller is: monitor. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 Replica is PRIMARY (1) >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 Instance name is server2. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 Current master is server2. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 DB_FAILOVER is ON. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: 2023/07/19 14:47:59 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: lease_expiry after monitor update: >Jul 19 14:47:59 ag(ag_cluster)[56056]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:48:00 ag(ag_cluster)[56056]: INFO: ag_cluster monitor : 8 >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: mssql_validate >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: Resource agent invoked with: notify >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: mssql_notify pre-stop >Jul 19 14:48:06 ag(ag_cluster)[56410]: INFO: ag_cluster notify : 0 >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: mssql_validate >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: Resource agent invoked with: notify >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: mssql_notify pre-stop >Jul 19 14:48:06 ag(ag_cluster)[56455]: INFO: ag_cluster notify : 0 >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: mssql_validate >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: Resource agent invoked with: notify >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: mssql_notify post-stop >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: notify: 2023/07/19 14:48:06 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: notify: 2023/07/19 14:48:06 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: notify: 2023/07/19 14:48:06 Replica is PRIMARY (1) >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: notify: 2023/07/19 14:48:06 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: notify: 2023/07/19 14:48:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:48:06 ag(ag_cluster)[56500]: INFO: ag_cluster notify : 0 >Jul 19 14:48:06 ag(ag_cluster)[56587]: INFO: mssql_validate >Jul 19 14:48:06 ag(ag_cluster)[56587]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:06 ag(ag_cluster)[56587]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: Resource agent invoked with: notify >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: mssql_notify pre-start >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 Replica is PRIMARY (1) >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: 2023/07/19 14:48:07 Sequence number is 384D:6 (61903363637254) >Jul 19 14:48:07 ag(ag_cluster)[56587]: INFO: notify: SEQUENCE_NUMBER: 61903363637254 >Jul 19 14:48:12 ag(ag_cluster)[56587]: INFO: ag_cluster notify : 0 >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: mssql_validate >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: Resource agent invoked with: monitor >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: mssql_monitor >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:48:12 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:12 Connected to the instance at localhost:1433 >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 Monitor Caller is: monitor. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 Replica is PRIMARY (1) >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 Instance name is server2. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 Current master is server2. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 DB_FAILOVER is ON. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: 2023/07/19 14:48:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: lease_expiry after monitor update: >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:48:17 ag(ag_cluster)[56700]: INFO: ag_cluster monitor : 8 >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: mssql_validate >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: Resource agent invoked with: notify >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: mssql_notify pre-start >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 Replica is PRIMARY (1) >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: 2023/07/19 14:48:17 Sequence number is 384D:7 (61903363637255) >Jul 19 14:48:17 ag(ag_cluster)[56927]: INFO: notify: SEQUENCE_NUMBER: 61903363637255 >Jul 19 14:48:22 ag(ag_cluster)[56927]: INFO: ag_cluster notify : 0 >Jul 19 14:48:22 ag(ag_cluster)[57040]: INFO: mssql_validate >Jul 19 14:48:22 ag(ag_cluster)[57040]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:22 ag(ag_cluster)[57040]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: Resource agent invoked with: notify >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: mssql_notify pre-start >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 Replica is PRIMARY (1) >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: 2023/07/19 14:48:23 Sequence number is 384D:7 (61903363637255) >Jul 19 14:48:23 ag(ag_cluster)[57040]: INFO: notify: SEQUENCE_NUMBER: 61903363637255 >Jul 19 14:48:28 ag(ag_cluster)[57040]: INFO: ag_cluster notify : 0 >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: mssql_validate >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: Resource agent invoked with: monitor >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: mssql_monitor >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:28 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:48:28 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:28 Connected to the instance at localhost:1433 >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Monitor Caller is: monitor. >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Replica is PRIMARY (1) >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Offlining replica... >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Replica is RESOLVING (0) >Jul 19 14:48:33 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Instance name is server2. >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Current master is server1 >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: server2. >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Setting the role to Secondary. >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Setting replica to SECONDARY role... >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: 2023/07/19 14:48:33 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: lease_expiry after monitor update: >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:48:34 ag(ag_cluster)[57195]: INFO: ag_cluster monitor : 0 >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: Resource agent invoked with: notify >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: mssql_notify post-start >Jul 19 14:48:34 ag(ag_cluster)[57391]: INFO: ag_cluster notify : 0 >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: Resource agent invoked with: notify >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: mssql_notify pre-demote >Jul 19 14:48:34 ag(ag_cluster)[57439]: INFO: ag_cluster notify : 0 >Jul 19 14:48:34 IPaddr2(virtualip)[57438]: INFO: IP status = ok, IP_CIP= >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: Resource agent invoked with: demote >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: mssql_demote >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: demote: 2023/07/19 14:48:34 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: demote: 2023/07/19 14:48:34 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: demote: 2023/07/19 14:48:34 Replica is RESOLVING (0) >Jul 19 14:48:34 ag(ag_cluster)[57537]: INFO: ag_cluster demote : 0 >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: Resource agent invoked with: notify >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: mssql_notify post-demote >Jul 19 14:48:34 ag(ag_cluster)[57630]: INFO: ag_cluster notify : 0 >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: Resource agent invoked with: notify >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: mssql_notify pre-stop >Jul 19 14:48:34 ag(ag_cluster)[57675]: INFO: ag_cluster notify : 0 >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: mssql_validate >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: Resource agent invoked with: stop >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: mssql_stop >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: stop: 2023/07/19 14:48:34 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: stop: 2023/07/19 14:48:34 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: stop: 2023/07/19 14:48:34 Replica is RESOLVING (0) >Jul 19 14:48:34 ag(ag_cluster)[57720]: INFO: ag_cluster stop : 0 >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: mssql_validate >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: Resource agent invoked with: start >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: mssql_start >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:45 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:45 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:48:45 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:45 Connected to the instance at localhost:1433 >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Replica is RESOLVING (0) >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Sequence number is 384D:7 (61903363637255) >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Line does not match expected syntax. Ignoring. >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61903363637255"] >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Line does not match expected syntax. Ignoring. >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 1 sequence numbers were found >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Max sequence number is 384D:7 (61903363637255) >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Monitor Caller is: start. >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Replica is RESOLVING (0) >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Instance name is server2. >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: 2023/07/19 14:48:50 Current master is . >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:48:50 ag(ag_cluster)[57814]: INFO: ag_cluster start : 0 >Jul 19 14:48:50 ag(ag_cluster)[58014]: INFO: mssql_validate >Jul 19 14:48:50 ag(ag_cluster)[58014]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:50 ag(ag_cluster)[58014]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:51 ag(ag_cluster)[58014]: INFO: Resource agent invoked with: notify >Jul 19 14:48:51 ag(ag_cluster)[58014]: INFO: mssql_notify post-start >Jul 19 14:48:51 ag(ag_cluster)[58014]: INFO: ag_cluster notify : 0 >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: mssql_validate >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: Resource agent invoked with: notify >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: mssql_notify pre-promote >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: notify: 2023/07/19 14:48:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: notify: 2023/07/19 14:48:51 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:48:51 ag(ag_cluster)[58060]: INFO: notify: 2023/07/19 14:48:51 Connected to the instance at localhost:1433 >Jul 19 14:48:56 ag(ag_cluster)[58060]: INFO: notify: 2023/07/19 14:48:56 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:48:56 ag(ag_cluster)[58060]: INFO: notify: 2023/07/19 14:48:56 Sequence number is 384D:7 (61903363637255) >Jul 19 14:48:56 ag(ag_cluster)[58060]: INFO: notify: SEQUENCE_NUMBER: 61903363637255 >Jul 19 14:49:01 ag(ag_cluster)[58060]: INFO: ag_cluster notify : 0 >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: mssql_validate >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: Resource agent invoked with: notify >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: mssql_notify pre-promote >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: notify: 2023/07/19 14:49:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: notify: 2023/07/19 14:49:01 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:49:01 ag(ag_cluster)[58167]: INFO: notify: 2023/07/19 14:49:01 Connected to the instance at localhost:1433 >Jul 19 14:49:06 ag(ag_cluster)[58167]: INFO: notify: 2023/07/19 14:49:06 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:49:06 ag(ag_cluster)[58167]: INFO: notify: 2023/07/19 14:49:06 Sequence number is 384D:7 (61903363637255) >Jul 19 14:49:06 ag(ag_cluster)[58167]: INFO: notify: SEQUENCE_NUMBER: 61903363637255 >Jul 19 14:49:11 ag(ag_cluster)[58167]: INFO: ag_cluster notify : 0 >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: mssql_validate >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: Resource agent invoked with: promote >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: mssql_promote >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:11 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:11 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:49:11 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:11 Connected to the instance at localhost:1433 >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Replica is RESOLVING (0) >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Line does not match expected syntax. Ignoring. >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61903363637255"] >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61903363637255"] >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 2 sequence numbers were found >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Max sequence number is 384D:7 (61903363637255) >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Sequence number of server2 is 384D:7 (61903363637255) >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Promoting replica to PRIMARY role... >Jul 19 14:49:16 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Replica is PRIMARY (1) >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Replica is now PRIMARY >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Monitor Caller is: promote. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Replica is PRIMARY (1) >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Instance name is server2. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Current master is server2. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 DB_FAILOVER is ON. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: 2023/07/19 14:49:16 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:49:17 ag(ag_cluster)[58276]: INFO: ag_cluster promote : 0 >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: mssql_validate >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: Resource agent invoked with: notify >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: mssql_notify post-promote >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: notify: 2023/07/19 14:49:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: notify: 2023/07/19 14:49:17 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: notify: 2023/07/19 14:49:17 Replica is PRIMARY (1) >Jul 19 14:49:17 ag(ag_cluster)[58551]: INFO: ag_cluster notify : 0 >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: mssql_validate >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: Resource agent invoked with: monitor >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: mssql_monitor >Jul 19 14:49:17 IPaddr2(virtualip)[58635]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:49:17 IPaddr2(virtualip)[58635]: INFO: Bringing device eth0 up >Jul 19 14:49:17 IPaddr2(virtualip)[58635]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:17 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:49:17 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:17 Connected to the instance at localhost:1433 >Jul 19 14:49:21 IPaddr2(virtualip)[58635]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 Monitor Caller is: monitor. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 Replica is PRIMARY (1) >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 Instance name is server2. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 Current master is server2. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 DB_FAILOVER is ON. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: 2023/07/19 14:49:22 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: lease_expiry after monitor update: >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:49:22 ag(ag_cluster)[58627]: INFO: ag_cluster monitor : 8 >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: mssql_validate >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: Resource agent invoked with: notify >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: mssql_notify pre-stop >Jul 19 14:49:29 ag(ag_cluster)[58982]: INFO: ag_cluster notify : 0 >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: mssql_validate >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: Resource agent invoked with: notify >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: mssql_notify pre-stop >Jul 19 14:49:29 ag(ag_cluster)[59027]: INFO: ag_cluster notify : 0 >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: mssql_validate >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: Resource agent invoked with: notify >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: mssql_notify post-stop >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: notify: 2023/07/19 14:49:29 Replica is PRIMARY (1) >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: notify: 2023/07/19 14:49:29 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: notify: 2023/07/19 14:49:29 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:29 ag(ag_cluster)[59072]: INFO: ag_cluster notify : 0 >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: mssql_validate >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: Resource agent invoked with: notify >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: mssql_notify pre-start >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 Replica is PRIMARY (1) >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: 2023/07/19 14:49:29 Sequence number is 384E:6 (61907658604550) >Jul 19 14:49:29 ag(ag_cluster)[59159]: INFO: notify: SEQUENCE_NUMBER: 61907658604550 >Jul 19 14:49:34 ag(ag_cluster)[59159]: INFO: ag_cluster notify : 0 >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: mssql_validate >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: Resource agent invoked with: monitor >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: mssql_monitor >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:49:35 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:35 Connected to the instance at localhost:1433 >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 Monitor Caller is: monitor. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 Replica is PRIMARY (1) >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 Instance name is server2. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 Current master is server2. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 DB_FAILOVER is ON. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: 2023/07/19 14:49:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: lease_expiry after monitor update: >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:49:40 ag(ag_cluster)[59271]: INFO: ag_cluster monitor : 8 >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: mssql_validate >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: Resource agent invoked with: notify >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: mssql_notify pre-start >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 Replica is PRIMARY (1) >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: 2023/07/19 14:49:40 Sequence number is 384E:B (61907658604555) >Jul 19 14:49:40 ag(ag_cluster)[59498]: INFO: notify: SEQUENCE_NUMBER: 61907658604555 >Jul 19 14:49:45 ag(ag_cluster)[59498]: INFO: ag_cluster notify : 0 >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: mssql_validate >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: Resource agent invoked with: notify >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: mssql_notify pre-start >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 Replica is PRIMARY (1) >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: 2023/07/19 14:49:45 Sequence number is 384E:B (61907658604555) >Jul 19 14:49:45 ag(ag_cluster)[59618]: INFO: notify: SEQUENCE_NUMBER: 61907658604555 >Jul 19 14:49:50 ag(ag_cluster)[59618]: INFO: ag_cluster notify : 0 >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: mssql_validate >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: Resource agent invoked with: monitor >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: mssql_monitor >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:51 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:51 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:49:51 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:51 Connected to the instance at localhost:1433 >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Monitor Caller is: monitor. >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Replica is PRIMARY (1) >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Offlining replica... >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Replica is RESOLVING (0) >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Instance name is server2. >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Current master is server1 >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: server2. >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Setting the role to Secondary. >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Setting replica to SECONDARY role... >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: 2023/07/19 14:49:56 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: lease_expiry after monitor update: >Jul 19 14:49:56 ag(ag_cluster)[59774]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:49:57 ag(ag_cluster)[59774]: INFO: ag_cluster monitor : 0 >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: Resource agent invoked with: notify >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: mssql_notify post-start >Jul 19 14:49:57 ag(ag_cluster)[59972]: INFO: ag_cluster notify : 0 >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: Resource agent invoked with: notify >Jul 19 14:49:57 IPaddr2(virtualip)[60018]: INFO: IP status = ok, IP_CIP= >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: mssql_notify pre-demote >Jul 19 14:49:57 ag(ag_cluster)[60019]: INFO: ag_cluster notify : 0 >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: Resource agent invoked with: demote >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: mssql_demote >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: demote: 2023/07/19 14:49:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: demote: 2023/07/19 14:49:57 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: demote: 2023/07/19 14:49:57 Replica is RESOLVING (0) >Jul 19 14:49:57 ag(ag_cluster)[60117]: INFO: ag_cluster demote : 0 >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: Resource agent invoked with: notify >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: mssql_notify post-demote >Jul 19 14:49:57 ag(ag_cluster)[60211]: INFO: ag_cluster notify : 0 >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: Resource agent invoked with: notify >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: mssql_notify pre-stop >Jul 19 14:49:57 ag(ag_cluster)[60256]: INFO: ag_cluster notify : 0 >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: mssql_validate >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: Resource agent invoked with: stop >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: mssql_stop >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: stop: 2023/07/19 14:49:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: stop: 2023/07/19 14:49:57 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: stop: 2023/07/19 14:49:57 Replica is RESOLVING (0) >Jul 19 14:49:57 ag(ag_cluster)[60301]: INFO: ag_cluster stop : 0 >Jul 19 14:50:08 ag(ag_cluster)[60388]: INFO: mssql_validate >Jul 19 14:50:08 ag(ag_cluster)[60388]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:08 ag(ag_cluster)[60388]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:08 ag(ag_cluster)[60388]: INFO: Resource agent invoked with: start >Jul 19 14:50:08 ag(ag_cluster)[60388]: INFO: mssql_start >Jul 19 14:50:09 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:08 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:50:09 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:08 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:50:09 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:08 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:09 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:09 Connected to the instance at localhost:1433 >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Replica is RESOLVING (0) >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Sequence number is 384E:B (61907658604555) >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Line does not match expected syntax. Ignoring. >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61907658604555"] >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Line does not match expected syntax. Ignoring. >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 1 sequence numbers were found >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Max sequence number is 384E:B (61907658604555) >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Monitor Caller is: start. >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Replica is RESOLVING (0) >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Instance name is server2. >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: 2023/07/19 14:50:14 Current master is . >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:50:14 ag(ag_cluster)[60388]: INFO: ag_cluster start : 0 >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: mssql_validate >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: Resource agent invoked with: notify >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: mssql_notify post-start >Jul 19 14:50:14 ag(ag_cluster)[60587]: INFO: ag_cluster notify : 0 >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: mssql_validate >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: Resource agent invoked with: notify >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: mssql_notify pre-promote >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: notify: 2023/07/19 14:50:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: notify: 2023/07/19 14:50:14 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:14 ag(ag_cluster)[60633]: INFO: notify: 2023/07/19 14:50:14 Connected to the instance at localhost:1433 >Jul 19 14:50:19 ag(ag_cluster)[60633]: INFO: notify: 2023/07/19 14:50:19 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:50:19 ag(ag_cluster)[60633]: INFO: notify: 2023/07/19 14:50:19 Sequence number is 384E:B (61907658604555) >Jul 19 14:50:19 ag(ag_cluster)[60633]: INFO: notify: SEQUENCE_NUMBER: 61907658604555 >Jul 19 14:50:24 ag(ag_cluster)[60633]: INFO: ag_cluster notify : 0 >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: mssql_validate >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: Resource agent invoked with: notify >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: mssql_notify pre-promote >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: notify: 2023/07/19 14:50:24 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: notify: 2023/07/19 14:50:24 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:24 ag(ag_cluster)[60739]: INFO: notify: 2023/07/19 14:50:24 Connected to the instance at localhost:1433 >Jul 19 14:50:29 ag(ag_cluster)[60739]: INFO: notify: 2023/07/19 14:50:29 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:50:30 ag(ag_cluster)[60739]: INFO: notify: 2023/07/19 14:50:29 Sequence number is 384E:B (61907658604555) >Jul 19 14:50:30 ag(ag_cluster)[60739]: INFO: notify: SEQUENCE_NUMBER: 61907658604555 >Jul 19 14:50:35 ag(ag_cluster)[60739]: INFO: ag_cluster notify : 0 >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: mssql_validate >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: Resource agent invoked with: promote >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: mssql_promote >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:35 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:35 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:35 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:35 Connected to the instance at localhost:1433 >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Replica is RESOLVING (0) >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Line does not match expected syntax. Ignoring. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61907658604555"] >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61907658604555"] >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 2 sequence numbers were found >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Max sequence number is 384E:B (61907658604555) >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Sequence number of server2 is 384E:B (61907658604555) >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Promoting replica to PRIMARY role... >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Replica is PRIMARY (1) >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Replica is now PRIMARY >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Monitor Caller is: promote. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Replica is PRIMARY (1) >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Instance name is server2. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Current master is server2. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 DB_FAILOVER is ON. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: 2023/07/19 14:50:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:50:40 ag(ag_cluster)[60848]: INFO: ag_cluster promote : 0 >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: mssql_validate >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: Resource agent invoked with: notify >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: mssql_notify post-promote >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: notify: 2023/07/19 14:50:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: notify: 2023/07/19 14:50:40 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: notify: 2023/07/19 14:50:40 Replica is PRIMARY (1) >Jul 19 14:50:40 ag(ag_cluster)[61120]: INFO: ag_cluster notify : 0 >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: mssql_validate >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: Resource agent invoked with: monitor >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: mssql_monitor >Jul 19 14:50:41 IPaddr2(virtualip)[61206]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:50:41 IPaddr2(virtualip)[61206]: INFO: Bringing device eth0 up >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:50:41 IPaddr2(virtualip)[61206]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:41 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:41 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:41 Connected to the instance at localhost:1433 >Jul 19 14:50:45 IPaddr2(virtualip)[61206]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 Monitor Caller is: monitor. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 Replica is PRIMARY (1) >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 Instance name is server2. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 Current master is server2. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 DB_FAILOVER is ON. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: 2023/07/19 14:50:46 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: lease_expiry after monitor update: >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:50:46 ag(ag_cluster)[61196]: INFO: ag_cluster monitor : 8 >Jul 19 14:50:52 ag(ag_cluster)[61555]: INFO: mssql_validate >Jul 19 14:50:52 ag(ag_cluster)[61555]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:52 ag(ag_cluster)[61555]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:52 ag(ag_cluster)[61555]: INFO: Resource agent invoked with: notify >Jul 19 14:50:52 ag(ag_cluster)[61555]: INFO: mssql_notify pre-stop >Jul 19 14:50:53 ag(ag_cluster)[61555]: INFO: ag_cluster notify : 0 >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: mssql_validate >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: Resource agent invoked with: notify >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: mssql_notify pre-stop >Jul 19 14:50:53 ag(ag_cluster)[61600]: INFO: ag_cluster notify : 0 >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: mssql_validate >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: Resource agent invoked with: notify >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: mssql_notify post-stop >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: notify: 2023/07/19 14:50:53 Replica is PRIMARY (1) >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: notify: 2023/07/19 14:50:53 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: notify: 2023/07/19 14:50:53 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:50:53 ag(ag_cluster)[61645]: INFO: ag_cluster notify : 0 >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: mssql_validate >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: Resource agent invoked with: notify >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: mssql_notify pre-start >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 Replica is PRIMARY (1) >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: 2023/07/19 14:50:53 Sequence number is 384F:6 (61911953571846) >Jul 19 14:50:53 ag(ag_cluster)[61732]: INFO: notify: SEQUENCE_NUMBER: 61911953571846 >Jul 19 14:50:58 ag(ag_cluster)[61732]: INFO: ag_cluster notify : 0 >Jul 19 14:50:58 ag(ag_cluster)[61843]: INFO: mssql_validate >Jul 19 14:50:58 ag(ag_cluster)[61843]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:50:58 ag(ag_cluster)[61843]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:50:58 ag(ag_cluster)[61843]: INFO: Resource agent invoked with: monitor >Jul 19 14:50:59 ag(ag_cluster)[61843]: INFO: mssql_monitor >Jul 19 14:50:59 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:50:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:50:59 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:50:59 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:50:59 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:50:59 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:50:59 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:50:59 Connected to the instance at localhost:1433 >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 Monitor Caller is: monitor. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 Replica is PRIMARY (1) >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 Instance name is server2. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 Current master is server2. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 DB_FAILOVER is ON. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: 2023/07/19 14:51:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: lease_expiry after monitor update: >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:51:04 ag(ag_cluster)[61843]: INFO: ag_cluster monitor : 8 >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: mssql_validate >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: Resource agent invoked with: notify >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: mssql_notify pre-start >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 Replica is PRIMARY (1) >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: 2023/07/19 14:51:04 Sequence number is 384F:B (61911953571851) >Jul 19 14:51:04 ag(ag_cluster)[62069]: INFO: notify: SEQUENCE_NUMBER: 61911953571851 >Jul 19 14:51:09 ag(ag_cluster)[62069]: INFO: ag_cluster notify : 0 >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: mssql_validate >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: Resource agent invoked with: notify >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: mssql_notify pre-start >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:51:09 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:10 Replica is PRIMARY (1) >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:10 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:10 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:10 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: 2023/07/19 14:51:10 Sequence number is 384F:B (61911953571851) >Jul 19 14:51:10 ag(ag_cluster)[62182]: INFO: notify: SEQUENCE_NUMBER: 61911953571851 >Jul 19 14:51:15 ag(ag_cluster)[62182]: INFO: ag_cluster notify : 0 >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: mssql_validate >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: Resource agent invoked with: monitor >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: mssql_monitor >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:15 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:51:15 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:15 Connected to the instance at localhost:1433 >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Monitor Caller is: monitor. >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Replica is PRIMARY (1) >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Offlining replica... >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Replica is RESOLVING (0) >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Instance name is server2. >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Current master is server1 >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: server2. >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Setting the role to Secondary. >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Setting replica to SECONDARY role... >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: 2023/07/19 14:51:20 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: lease_expiry after monitor update: >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:51:20 ag(ag_cluster)[62335]: INFO: ag_cluster monitor : 0 >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: mssql_validate >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: Resource agent invoked with: notify >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: mssql_notify post-start >Jul 19 14:51:20 ag(ag_cluster)[62531]: INFO: ag_cluster notify : 0 >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: mssql_validate >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: Resource agent invoked with: notify >Jul 19 14:51:21 IPaddr2(virtualip)[62577]: INFO: IP status = ok, IP_CIP= >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: mssql_notify pre-demote >Jul 19 14:51:21 ag(ag_cluster)[62578]: INFO: ag_cluster notify : 0 >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: mssql_validate >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: Resource agent invoked with: demote >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: mssql_demote >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: demote: 2023/07/19 14:51:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: demote: 2023/07/19 14:51:21 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: demote: 2023/07/19 14:51:21 Replica is RESOLVING (0) >Jul 19 14:51:21 ag(ag_cluster)[62676]: INFO: ag_cluster demote : 0 >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: mssql_validate >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: Resource agent invoked with: notify >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: mssql_notify post-demote >Jul 19 14:51:21 ag(ag_cluster)[62769]: INFO: ag_cluster notify : 0 >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: mssql_validate >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: Resource agent invoked with: notify >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: mssql_notify pre-stop >Jul 19 14:51:21 ag(ag_cluster)[62814]: INFO: ag_cluster notify : 0 >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: mssql_validate >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: Resource agent invoked with: stop >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: mssql_stop >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: stop: 2023/07/19 14:51:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: stop: 2023/07/19 14:51:21 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: stop: 2023/07/19 14:51:21 Replica is RESOLVING (0) >Jul 19 14:51:21 ag(ag_cluster)[62859]: INFO: ag_cluster stop : 0 >Jul 19 14:51:31 ag(ag_cluster)[62941]: INFO: mssql_validate >Jul 19 14:51:31 ag(ag_cluster)[62941]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:31 ag(ag_cluster)[62941]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: Resource agent invoked with: start >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: mssql_start >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:32 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:32 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:51:32 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:32 Connected to the instance at localhost:1433 >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Replica is RESOLVING (0) >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Sequence number is 384F:B (61911953571851) >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Line does not match expected syntax. Ignoring. >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61911953571851"] >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Line does not match expected syntax. Ignoring. >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 1 sequence numbers were found >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Max sequence number is 384F:B (61911953571851) >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Monitor Caller is: start. >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Replica is RESOLVING (0) >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Instance name is server2. >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: 2023/07/19 14:51:37 Current master is . >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:51:37 ag(ag_cluster)[62941]: INFO: ag_cluster start : 0 >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: mssql_validate >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: Resource agent invoked with: notify >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: mssql_notify post-start >Jul 19 14:51:37 ag(ag_cluster)[63138]: INFO: ag_cluster notify : 0 >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: mssql_validate >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: Resource agent invoked with: notify >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: mssql_notify pre-promote >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: notify: 2023/07/19 14:51:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: notify: 2023/07/19 14:51:37 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:51:37 ag(ag_cluster)[63184]: INFO: notify: 2023/07/19 14:51:37 Connected to the instance at localhost:1433 >Jul 19 14:51:42 ag(ag_cluster)[63184]: INFO: notify: 2023/07/19 14:51:42 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:51:42 ag(ag_cluster)[63184]: INFO: notify: 2023/07/19 14:51:42 Sequence number is 384F:B (61911953571851) >Jul 19 14:51:42 ag(ag_cluster)[63184]: INFO: notify: SEQUENCE_NUMBER: 61911953571851 >Jul 19 14:51:47 ag(ag_cluster)[63184]: INFO: ag_cluster notify : 0 >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: mssql_validate >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: Resource agent invoked with: notify >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: mssql_notify pre-promote >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: notify: 2023/07/19 14:51:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: notify: 2023/07/19 14:51:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:51:48 ag(ag_cluster)[63296]: INFO: notify: 2023/07/19 14:51:48 Connected to the instance at localhost:1433 >Jul 19 14:51:53 ag(ag_cluster)[63296]: INFO: notify: 2023/07/19 14:51:53 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:51:53 ag(ag_cluster)[63296]: INFO: notify: 2023/07/19 14:51:53 Sequence number is 384F:B (61911953571851) >Jul 19 14:51:53 ag(ag_cluster)[63296]: INFO: notify: SEQUENCE_NUMBER: 61911953571851 >Jul 19 14:51:58 ag(ag_cluster)[63296]: INFO: ag_cluster notify : 0 >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: mssql_validate >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: Resource agent invoked with: promote >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: mssql_promote >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:51:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:51:58 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:51:58 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:51:58 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:51:58 Connected to the instance at localhost:1433 >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Replica is RESOLVING (0) >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Line does not match expected syntax. Ignoring. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61911953571851"] >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61911953571851"] >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 2 sequence numbers were found >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Max sequence number is 384F:B (61911953571851) >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Sequence number of server2 is 384F:B (61911953571851) >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Promoting replica to PRIMARY role... >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Replica is PRIMARY (1) >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Replica is now PRIMARY >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Monitor Caller is: promote. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Replica is PRIMARY (1) >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Instance name is server2. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Current master is server2. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 DB_FAILOVER is ON. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: 2023/07/19 14:52:03 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:52:03 ag(ag_cluster)[63404]: INFO: ag_cluster promote : 0 >Jul 19 14:52:03 ag(ag_cluster)[63678]: INFO: mssql_validate >Jul 19 14:52:03 ag(ag_cluster)[63678]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:03 ag(ag_cluster)[63678]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:03 ag(ag_cluster)[63678]: INFO: Resource agent invoked with: notify >Jul 19 14:52:04 ag(ag_cluster)[63678]: INFO: mssql_notify post-promote >Jul 19 14:52:04 ag(ag_cluster)[63678]: INFO: notify: 2023/07/19 14:52:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:52:04 ag(ag_cluster)[63678]: INFO: notify: 2023/07/19 14:52:04 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:04 ag(ag_cluster)[63678]: INFO: notify: 2023/07/19 14:52:04 Replica is PRIMARY (1) >Jul 19 14:52:04 ag(ag_cluster)[63678]: INFO: ag_cluster notify : 0 >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: mssql_validate >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: Resource agent invoked with: monitor >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: mssql_monitor >Jul 19 14:52:04 IPaddr2(virtualip)[63763]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:52:04 IPaddr2(virtualip)[63763]: INFO: Bringing device eth0 up >Jul 19 14:52:04 IPaddr2(virtualip)[63763]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:52:04 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:04 Connected to the instance at localhost:1433 >Jul 19 14:52:08 IPaddr2(virtualip)[63763]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 Monitor Caller is: monitor. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 Replica is PRIMARY (1) >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 Instance name is server2. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 Current master is server2. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 DB_FAILOVER is ON. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: 2023/07/19 14:52:09 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: lease_expiry after monitor update: >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:52:09 ag(ag_cluster)[63753]: INFO: ag_cluster monitor : 8 >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: mssql_validate >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: Resource agent invoked with: notify >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: mssql_notify pre-stop >Jul 19 14:52:17 ag(ag_cluster)[64108]: INFO: ag_cluster notify : 0 >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: mssql_validate >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: Resource agent invoked with: notify >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: mssql_notify pre-stop >Jul 19 14:52:17 ag(ag_cluster)[64153]: INFO: ag_cluster notify : 0 >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: mssql_validate >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: Resource agent invoked with: notify >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: mssql_notify post-stop >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: notify: 2023/07/19 14:52:17 Replica is PRIMARY (1) >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: notify: 2023/07/19 14:52:17 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: notify: 2023/07/19 14:52:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:17 ag(ag_cluster)[64198]: INFO: ag_cluster notify : 0 >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: mssql_validate >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: Resource agent invoked with: notify >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: mssql_notify pre-start >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 Replica is PRIMARY (1) >Jul 19 14:52:17 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:18 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:18 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:52:18 ag(ag_cluster)[64285]: INFO: notify: 2023/07/19 14:52:17 Sequence number is 3850:8 (61916248539144) >Jul 19 14:52:18 ag(ag_cluster)[64285]: INFO: notify: SEQUENCE_NUMBER: 61916248539144 >Jul 19 14:52:23 ag(ag_cluster)[64285]: INFO: ag_cluster notify : 0 >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: mssql_validate >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: Resource agent invoked with: monitor >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: mssql_monitor >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:23 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:23 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:52:23 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:23 Connected to the instance at localhost:1433 >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 Monitor Caller is: monitor. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 Replica is PRIMARY (1) >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 Instance name is server2. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 Current master is server2. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 DB_FAILOVER is ON. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: 2023/07/19 14:52:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: lease_expiry after monitor update: >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:52:28 ag(ag_cluster)[64398]: INFO: ag_cluster monitor : 8 >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: mssql_validate >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: Resource agent invoked with: notify >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: mssql_notify pre-start >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 Replica is PRIMARY (1) >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: 2023/07/19 14:52:28 Sequence number is 3850:B (61916248539147) >Jul 19 14:52:28 ag(ag_cluster)[64624]: INFO: notify: SEQUENCE_NUMBER: 61916248539147 >Jul 19 14:52:33 ag(ag_cluster)[64624]: INFO: ag_cluster notify : 0 >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: mssql_validate >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: Resource agent invoked with: notify >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: mssql_notify pre-start >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 Replica is PRIMARY (1) >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: 2023/07/19 14:52:33 Sequence number is 3850:B (61916248539147) >Jul 19 14:52:33 ag(ag_cluster)[64735]: INFO: notify: SEQUENCE_NUMBER: 61916248539147 >Jul 19 14:52:38 ag(ag_cluster)[64735]: INFO: ag_cluster notify : 0 >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: mssql_validate >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: Resource agent invoked with: monitor >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: mssql_monitor >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:39 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:39 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:39 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:52:39 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:39 Connected to the instance at localhost:1433 >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Monitor Caller is: monitor. >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Replica is PRIMARY (1) >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Offlining replica... >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Replica is RESOLVING (0) >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Instance name is server2. >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Current master is server1 >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: server2. >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Setting the role to Secondary. >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Setting replica to SECONDARY role... >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: 2023/07/19 14:52:44 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:52:44 ag(ag_cluster)[64889]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:52:45 ag(ag_cluster)[64889]: INFO: lease_expiry after monitor update: >Jul 19 14:52:45 ag(ag_cluster)[64889]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:52:45 ag(ag_cluster)[64889]: INFO: ag_cluster monitor : 0 >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: Resource agent invoked with: notify >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: mssql_notify post-start >Jul 19 14:52:45 ag(ag_cluster)[65127]: INFO: ag_cluster notify : 0 >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: Resource agent invoked with: notify >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: mssql_notify pre-demote >Jul 19 14:52:45 IPaddr2(virtualip)[65181]: INFO: IP status = ok, IP_CIP= >Jul 19 14:52:45 ag(ag_cluster)[65182]: INFO: ag_cluster notify : 0 >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: Resource agent invoked with: demote >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: mssql_demote >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: demote: 2023/07/19 14:52:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: demote: 2023/07/19 14:52:45 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: demote: 2023/07/19 14:52:45 Replica is RESOLVING (0) >Jul 19 14:52:45 ag(ag_cluster)[65280]: INFO: ag_cluster demote : 0 >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: Resource agent invoked with: notify >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: mssql_notify post-demote >Jul 19 14:52:45 ag(ag_cluster)[65373]: INFO: ag_cluster notify : 0 >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: Resource agent invoked with: notify >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: mssql_notify pre-stop >Jul 19 14:52:45 ag(ag_cluster)[65418]: INFO: ag_cluster notify : 0 >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: mssql_validate >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: Resource agent invoked with: stop >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: mssql_stop >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: stop: 2023/07/19 14:52:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:52:45 ag(ag_cluster)[65463]: INFO: stop: 2023/07/19 14:52:45 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:52:46 ag(ag_cluster)[65463]: INFO: stop: 2023/07/19 14:52:45 Replica is RESOLVING (0) >Jul 19 14:52:46 ag(ag_cluster)[65463]: INFO: ag_cluster stop : 0 >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: mssql_validate >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: Resource agent invoked with: start >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: mssql_start >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:52:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:52:57 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:52:57 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:52:57 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:52:57 Connected to the instance at localhost:1433 >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Replica is RESOLVING (0) >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Sequence number is 3850:B (61916248539147) >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Line does not match expected syntax. Ignoring. >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61916248539147"] >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Line does not match expected syntax. Ignoring. >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 1 sequence numbers were found >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Max sequence number is 3850:B (61916248539147) >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Monitor Caller is: start. >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Replica is RESOLVING (0) >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Instance name is server2. >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: 2023/07/19 14:53:02 Current master is . >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:53:02 ag(ag_cluster)[65545]: INFO: ag_cluster start : 0 >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: mssql_validate >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: Resource agent invoked with: notify >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: mssql_notify post-start >Jul 19 14:53:02 ag(ag_cluster)[65744]: INFO: ag_cluster notify : 0 >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: mssql_validate >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: Resource agent invoked with: notify >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: mssql_notify pre-promote >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: notify: 2023/07/19 14:53:02 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: notify: 2023/07/19 14:53:02 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:53:02 ag(ag_cluster)[65790]: INFO: notify: 2023/07/19 14:53:02 Connected to the instance at localhost:1433 >Jul 19 14:53:07 ag(ag_cluster)[65790]: INFO: notify: 2023/07/19 14:53:07 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:53:07 ag(ag_cluster)[65790]: INFO: notify: 2023/07/19 14:53:07 Sequence number is 3850:B (61916248539147) >Jul 19 14:53:07 ag(ag_cluster)[65790]: INFO: notify: SEQUENCE_NUMBER: 61916248539147 >Jul 19 14:53:13 ag(ag_cluster)[65790]: INFO: ag_cluster notify : 0 >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: mssql_validate >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: Resource agent invoked with: notify >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: mssql_notify pre-promote >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: notify: 2023/07/19 14:53:13 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: notify: 2023/07/19 14:53:13 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:53:13 ag(ag_cluster)[65894]: INFO: notify: 2023/07/19 14:53:13 Connected to the instance at localhost:1433 >Jul 19 14:53:18 ag(ag_cluster)[65894]: INFO: notify: 2023/07/19 14:53:18 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:53:18 ag(ag_cluster)[65894]: INFO: notify: 2023/07/19 14:53:18 Sequence number is 3850:B (61916248539147) >Jul 19 14:53:18 ag(ag_cluster)[65894]: INFO: notify: SEQUENCE_NUMBER: 61916248539147 >Jul 19 14:53:23 ag(ag_cluster)[65894]: INFO: ag_cluster notify : 0 >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: mssql_validate >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: Resource agent invoked with: promote >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: mssql_promote >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:23 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:23 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:53:23 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:23 Connected to the instance at localhost:1433 >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Replica is RESOLVING (0) >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Line does not match expected syntax. Ignoring. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61916248539147"] >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61916248539147"] >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 2 sequence numbers were found >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Max sequence number is 3850:B (61916248539147) >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Sequence number of server2 is 3850:B (61916248539147) >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Promoting replica to PRIMARY role... >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Replica is PRIMARY (1) >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Replica is now PRIMARY >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Monitor Caller is: promote. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Replica is PRIMARY (1) >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Instance name is server2. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Current master is server2. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 DB_FAILOVER is ON. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: 2023/07/19 14:53:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:53:28 ag(ag_cluster)[65998]: INFO: ag_cluster promote : 0 >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: mssql_validate >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: Resource agent invoked with: notify >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: mssql_notify post-promote >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: notify: 2023/07/19 14:53:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: notify: 2023/07/19 14:53:29 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: notify: 2023/07/19 14:53:29 Replica is PRIMARY (1) >Jul 19 14:53:29 ag(ag_cluster)[66270]: INFO: ag_cluster notify : 0 >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: mssql_validate >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: Resource agent invoked with: monitor >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: mssql_monitor >Jul 19 14:53:29 IPaddr2(virtualip)[66360]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:53:29 IPaddr2(virtualip)[66360]: INFO: Bringing device eth0 up >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:53:29 IPaddr2(virtualip)[66360]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:29 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:53:29 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:29 Connected to the instance at localhost:1433 >Jul 19 14:53:33 IPaddr2(virtualip)[66360]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 Monitor Caller is: monitor. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 Replica is PRIMARY (1) >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 Instance name is server2. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 Current master is server2. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 DB_FAILOVER is ON. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: 2023/07/19 14:53:34 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: lease_expiry after monitor update: >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:53:34 ag(ag_cluster)[66345]: INFO: ag_cluster monitor : 8 >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: mssql_validate >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: Resource agent invoked with: notify >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: mssql_notify pre-stop >Jul 19 14:53:41 ag(ag_cluster)[66700]: INFO: ag_cluster notify : 0 >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: mssql_validate >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: Resource agent invoked with: notify >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: mssql_notify pre-stop >Jul 19 14:53:41 ag(ag_cluster)[66746]: INFO: ag_cluster notify : 0 >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: mssql_validate >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: Resource agent invoked with: notify >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: mssql_notify post-stop >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: notify: 2023/07/19 14:53:41 Replica is PRIMARY (1) >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: notify: 2023/07/19 14:53:41 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: notify: 2023/07/19 14:53:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:41 ag(ag_cluster)[66791]: INFO: ag_cluster notify : 0 >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: mssql_validate >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: Resource agent invoked with: notify >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: mssql_notify pre-start >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 Replica is PRIMARY (1) >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: 2023/07/19 14:53:41 Sequence number is 3851:6 (61920543506438) >Jul 19 14:53:41 ag(ag_cluster)[66877]: INFO: notify: SEQUENCE_NUMBER: 61920543506438 >Jul 19 14:53:47 ag(ag_cluster)[66877]: INFO: ag_cluster notify : 0 >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: mssql_validate >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: Resource agent invoked with: monitor >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: mssql_monitor >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:53:47 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:47 Connected to the instance at localhost:1433 >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 Monitor Caller is: monitor. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 Replica is PRIMARY (1) >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 Instance name is server2. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 Current master is server2. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 DB_FAILOVER is ON. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: 2023/07/19 14:53:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: lease_expiry after monitor update: >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:53:52 ag(ag_cluster)[66996]: INFO: ag_cluster monitor : 8 >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: mssql_validate >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: Resource agent invoked with: notify >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: mssql_notify pre-start >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 Replica is PRIMARY (1) >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: 2023/07/19 14:53:52 Sequence number is 3851:7 (61920543506439) >Jul 19 14:53:52 ag(ag_cluster)[67223]: INFO: notify: SEQUENCE_NUMBER: 61920543506439 >Jul 19 14:53:57 ag(ag_cluster)[67223]: INFO: ag_cluster notify : 0 >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: mssql_validate >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: Resource agent invoked with: notify >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: mssql_notify pre-start >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 Replica is PRIMARY (1) >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: 2023/07/19 14:53:57 Sequence number is 3851:7 (61920543506439) >Jul 19 14:53:57 ag(ag_cluster)[67337]: INFO: notify: SEQUENCE_NUMBER: 61920543506439 >Jul 19 14:54:02 ag(ag_cluster)[67337]: INFO: ag_cluster notify : 0 >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: mssql_validate >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: Resource agent invoked with: monitor >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: mssql_monitor >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:03 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:03 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:03 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:03 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:03 Connected to the instance at localhost:1433 >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Monitor Caller is: monitor. >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Replica is PRIMARY (1) >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Offlining replica... >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Replica is RESOLVING (0) >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Instance name is server2. >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Current master is server1 >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: server2. >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Setting the role to Secondary. >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Setting replica to SECONDARY role... >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: 2023/07/19 14:54:08 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: lease_expiry after monitor update: >Jul 19 14:54:08 ag(ag_cluster)[67492]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:54:09 ag(ag_cluster)[67492]: INFO: ag_cluster monitor : 0 >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: Resource agent invoked with: notify >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: mssql_notify post-start >Jul 19 14:54:09 ag(ag_cluster)[67690]: INFO: ag_cluster notify : 0 >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: Resource agent invoked with: notify >Jul 19 14:54:09 IPaddr2(virtualip)[67736]: INFO: IP status = ok, IP_CIP= >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: mssql_notify pre-demote >Jul 19 14:54:09 ag(ag_cluster)[67737]: INFO: ag_cluster notify : 0 >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: Resource agent invoked with: demote >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: mssql_demote >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: demote: 2023/07/19 14:54:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: demote: 2023/07/19 14:54:09 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: demote: 2023/07/19 14:54:09 Replica is RESOLVING (0) >Jul 19 14:54:09 ag(ag_cluster)[67835]: INFO: ag_cluster demote : 0 >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: Resource agent invoked with: notify >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: mssql_notify post-demote >Jul 19 14:54:09 ag(ag_cluster)[67928]: INFO: ag_cluster notify : 0 >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: Resource agent invoked with: notify >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: mssql_notify pre-stop >Jul 19 14:54:09 ag(ag_cluster)[67973]: INFO: ag_cluster notify : 0 >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: mssql_validate >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: Resource agent invoked with: stop >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: mssql_stop >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: stop: 2023/07/19 14:54:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: stop: 2023/07/19 14:54:09 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: stop: 2023/07/19 14:54:09 Replica is RESOLVING (0) >Jul 19 14:54:09 ag(ag_cluster)[68018]: INFO: ag_cluster stop : 0 >Jul 19 14:54:20 ag(ag_cluster)[68100]: INFO: mssql_validate >Jul 19 14:54:20 ag(ag_cluster)[68100]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:20 ag(ag_cluster)[68100]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:20 ag(ag_cluster)[68100]: INFO: Resource agent invoked with: start >Jul 19 14:54:21 ag(ag_cluster)[68100]: INFO: mssql_start >Jul 19 14:54:21 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:54:21 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:21 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:54:21 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:21 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:21 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:21 Connected to the instance at localhost:1433 >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Replica is RESOLVING (0) >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Sequence number is 3851:7 (61920543506439) >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Line does not match expected syntax. Ignoring. >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61920543506439"] >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Line does not match expected syntax. Ignoring. >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 1 sequence numbers were found >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Max sequence number is 3851:7 (61920543506439) >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Monitor Caller is: start. >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Replica is RESOLVING (0) >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Instance name is server2. >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: 2023/07/19 14:54:26 Current master is . >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:54:26 ag(ag_cluster)[68100]: INFO: ag_cluster start : 0 >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: mssql_validate >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: Resource agent invoked with: notify >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: mssql_notify post-start >Jul 19 14:54:26 ag(ag_cluster)[68299]: INFO: ag_cluster notify : 0 >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: mssql_validate >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: Resource agent invoked with: notify >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: mssql_notify pre-promote >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: notify: 2023/07/19 14:54:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: notify: 2023/07/19 14:54:26 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:26 ag(ag_cluster)[68345]: INFO: notify: 2023/07/19 14:54:26 Connected to the instance at localhost:1433 >Jul 19 14:54:31 ag(ag_cluster)[68345]: INFO: notify: 2023/07/19 14:54:31 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:54:31 ag(ag_cluster)[68345]: INFO: notify: 2023/07/19 14:54:31 Sequence number is 3851:7 (61920543506439) >Jul 19 14:54:31 ag(ag_cluster)[68345]: INFO: notify: SEQUENCE_NUMBER: 61920543506439 >Jul 19 14:54:36 ag(ag_cluster)[68345]: INFO: ag_cluster notify : 0 >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: mssql_validate >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: Resource agent invoked with: notify >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: mssql_notify pre-promote >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: notify: 2023/07/19 14:54:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: notify: 2023/07/19 14:54:36 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:36 ag(ag_cluster)[68448]: INFO: notify: 2023/07/19 14:54:36 Connected to the instance at localhost:1433 >Jul 19 14:54:41 ag(ag_cluster)[68448]: INFO: notify: 2023/07/19 14:54:41 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:54:41 ag(ag_cluster)[68448]: INFO: notify: 2023/07/19 14:54:41 Sequence number is 3851:7 (61920543506439) >Jul 19 14:54:41 ag(ag_cluster)[68448]: INFO: notify: SEQUENCE_NUMBER: 61920543506439 >Jul 19 14:54:46 ag(ag_cluster)[68448]: INFO: ag_cluster notify : 0 >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: mssql_validate >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: Resource agent invoked with: promote >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: mssql_promote >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:47 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:47 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:47 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:47 Connected to the instance at localhost:1433 >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Replica is RESOLVING (0) >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Line does not match expected syntax. Ignoring. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61920543506439"] >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61920543506439"] >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 2 sequence numbers were found >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Max sequence number is 3851:7 (61920543506439) >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Sequence number of server2 is 3851:7 (61920543506439) >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Promoting replica to PRIMARY role... >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Replica is PRIMARY (1) >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Replica is now PRIMARY >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Monitor Caller is: promote. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Replica is PRIMARY (1) >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Instance name is server2. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Current master is server2. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 DB_FAILOVER is ON. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: 2023/07/19 14:54:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:54:52 ag(ag_cluster)[68559]: INFO: ag_cluster promote : 0 >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: mssql_validate >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: Resource agent invoked with: notify >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: mssql_notify post-promote >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: notify: 2023/07/19 14:54:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: notify: 2023/07/19 14:54:52 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: notify: 2023/07/19 14:54:52 Replica is PRIMARY (1) >Jul 19 14:54:52 ag(ag_cluster)[68830]: INFO: ag_cluster notify : 0 >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: mssql_validate >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: Resource agent invoked with: monitor >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: mssql_monitor >Jul 19 14:54:52 IPaddr2(virtualip)[68916]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:54:52 IPaddr2(virtualip)[68916]: INFO: Bringing device eth0 up >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:54:52 IPaddr2(virtualip)[68916]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:54:52 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:52 Connected to the instance at localhost:1433 >Jul 19 14:54:56 IPaddr2(virtualip)[68916]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 Monitor Caller is: monitor. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 Replica is PRIMARY (1) >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 Instance name is server2. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 Current master is server2. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 DB_FAILOVER is ON. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: 2023/07/19 14:54:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: lease_expiry after monitor update: >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:54:57 ag(ag_cluster)[68905]: INFO: ag_cluster monitor : 8 >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: mssql_validate >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: Resource agent invoked with: notify >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: mssql_notify pre-stop >Jul 19 14:55:04 ag(ag_cluster)[69256]: INFO: ag_cluster notify : 0 >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: mssql_validate >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: Resource agent invoked with: notify >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: mssql_notify pre-stop >Jul 19 14:55:04 ag(ag_cluster)[69301]: INFO: ag_cluster notify : 0 >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: mssql_validate >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: Resource agent invoked with: notify >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: mssql_notify post-stop >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: notify: 2023/07/19 14:55:04 Replica is PRIMARY (1) >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: notify: 2023/07/19 14:55:04 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: notify: 2023/07/19 14:55:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:55:04 ag(ag_cluster)[69346]: INFO: ag_cluster notify : 0 >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: mssql_validate >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: Resource agent invoked with: notify >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: mssql_notify pre-start >Jul 19 14:55:04 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:05 Replica is PRIMARY (1) >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:05 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:05 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:05 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: 2023/07/19 14:55:05 Sequence number is 3852:6 (61924838473734) >Jul 19 14:55:05 ag(ag_cluster)[69433]: INFO: notify: SEQUENCE_NUMBER: 61924838473734 >Jul 19 14:55:10 ag(ag_cluster)[69433]: INFO: ag_cluster notify : 0 >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: mssql_validate >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: Resource agent invoked with: monitor >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: mssql_monitor >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:10 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:10 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:10 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:55:10 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:10 Connected to the instance at localhost:1433 >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 Monitor Caller is: monitor. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 Replica is PRIMARY (1) >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 Instance name is server2. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 Current master is server2. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 DB_FAILOVER is ON. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: 2023/07/19 14:55:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: lease_expiry after monitor update: >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:55:15 ag(ag_cluster)[69545]: INFO: ag_cluster monitor : 8 >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: mssql_validate >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: Resource agent invoked with: notify >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: mssql_notify pre-start >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 Replica is PRIMARY (1) >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: 2023/07/19 14:55:15 Sequence number is 3852:B (61924838473739) >Jul 19 14:55:15 ag(ag_cluster)[69773]: INFO: notify: SEQUENCE_NUMBER: 61924838473739 >Jul 19 14:55:20 ag(ag_cluster)[69773]: INFO: ag_cluster notify : 0 >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: mssql_validate >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: Resource agent invoked with: notify >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: mssql_notify pre-start >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 Replica is PRIMARY (1) >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: 2023/07/19 14:55:21 Sequence number is 3852:B (61924838473739) >Jul 19 14:55:21 ag(ag_cluster)[69884]: INFO: notify: SEQUENCE_NUMBER: 61924838473739 >Jul 19 14:55:26 ag(ag_cluster)[69884]: INFO: ag_cluster notify : 0 >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: mssql_validate >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: Resource agent invoked with: monitor >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: mssql_monitor >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:55:26 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:26 Connected to the instance at localhost:1433 >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Monitor Caller is: monitor. >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Replica is PRIMARY (1) >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Offlining replica... >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Replica is RESOLVING (0) >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Instance name is server2. >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Current master is server1 >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: server2. >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Setting the role to Secondary. >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Setting replica to SECONDARY role... >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: 2023/07/19 14:55:31 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: lease_expiry after monitor update: >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:55:31 ag(ag_cluster)[70038]: INFO: ag_cluster monitor : 0 >Jul 19 14:55:31 ag(ag_cluster)[70233]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70233]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70233]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:32 ag(ag_cluster)[70233]: INFO: Resource agent invoked with: notify >Jul 19 14:55:32 ag(ag_cluster)[70233]: INFO: mssql_notify post-start >Jul 19 14:55:32 ag(ag_cluster)[70233]: INFO: ag_cluster notify : 0 >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: Resource agent invoked with: notify >Jul 19 14:55:32 IPaddr2(virtualip)[70279]: INFO: IP status = ok, IP_CIP= >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: mssql_notify pre-demote >Jul 19 14:55:32 ag(ag_cluster)[70280]: INFO: ag_cluster notify : 0 >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: Resource agent invoked with: demote >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: mssql_demote >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: demote: 2023/07/19 14:55:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: demote: 2023/07/19 14:55:32 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: demote: 2023/07/19 14:55:32 Replica is RESOLVING (0) >Jul 19 14:55:32 ag(ag_cluster)[70378]: INFO: ag_cluster demote : 0 >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: Resource agent invoked with: notify >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: mssql_notify post-demote >Jul 19 14:55:32 ag(ag_cluster)[70471]: INFO: ag_cluster notify : 0 >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: Resource agent invoked with: notify >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: mssql_notify pre-stop >Jul 19 14:55:32 ag(ag_cluster)[70516]: INFO: ag_cluster notify : 0 >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: mssql_validate >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: Resource agent invoked with: stop >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: mssql_stop >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: stop: 2023/07/19 14:55:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: stop: 2023/07/19 14:55:32 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: stop: 2023/07/19 14:55:32 Replica is RESOLVING (0) >Jul 19 14:55:32 ag(ag_cluster)[70561]: INFO: ag_cluster stop : 0 >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: mssql_validate >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: Resource agent invoked with: start >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: mssql_start >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start] >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:43 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:43 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:55:43 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:43 Connected to the instance at localhost:1433 >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Replica is RESOLVING (0) >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Sequence number is 3852:B (61924838473739) >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Line does not match expected syntax. Ignoring. >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61924838473739"] >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""] >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Line does not match expected syntax. Ignoring. >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 1 sequence numbers were found >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Max sequence number is 3852:B (61924838473739) >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Monitor Caller is: start. >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Replica is RESOLVING (0) >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Instance name is server2. >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: 2023/07/19 14:55:48 Current master is . >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: start: PROMOTION_SCORE: 10 >Jul 19 14:55:48 ag(ag_cluster)[70643]: INFO: ag_cluster start : 0 >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: mssql_validate >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: Resource agent invoked with: notify >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: mssql_notify post-start >Jul 19 14:55:48 ag(ag_cluster)[70849]: INFO: ag_cluster notify : 0 >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: mssql_validate >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: Resource agent invoked with: notify >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: mssql_notify pre-promote >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: notify: 2023/07/19 14:55:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: notify: 2023/07/19 14:55:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:55:48 ag(ag_cluster)[70895]: INFO: notify: 2023/07/19 14:55:48 Connected to the instance at localhost:1433 >Jul 19 14:55:53 ag(ag_cluster)[70895]: INFO: notify: 2023/07/19 14:55:53 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:55:53 ag(ag_cluster)[70895]: INFO: notify: 2023/07/19 14:55:53 Sequence number is 3852:B (61924838473739) >Jul 19 14:55:53 ag(ag_cluster)[70895]: INFO: notify: SEQUENCE_NUMBER: 61924838473739 >Jul 19 14:55:58 ag(ag_cluster)[70895]: INFO: ag_cluster notify : 0 >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: mssql_validate >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: Resource agent invoked with: notify >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: mssql_notify pre-promote >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: notify: 2023/07/19 14:55:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote] >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: notify: 2023/07/19 14:55:58 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:55:58 ag(ag_cluster)[70999]: INFO: notify: 2023/07/19 14:55:58 Connected to the instance at localhost:1433 >Jul 19 14:56:03 ag(ag_cluster)[70999]: INFO: notify: 2023/07/19 14:56:03 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:56:04 ag(ag_cluster)[70999]: INFO: notify: 2023/07/19 14:56:03 Sequence number is 3852:B (61924838473739) >Jul 19 14:56:04 ag(ag_cluster)[70999]: INFO: notify: SEQUENCE_NUMBER: 61924838473739 >Jul 19 14:56:09 ag(ag_cluster)[70999]: INFO: ag_cluster notify : 0 >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: mssql_validate >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: Resource agent invoked with: promote >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: mssql_promote >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote] >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:09 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62] >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:09 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:56:09 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:09 Connected to the instance at localhost:1433 >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Replica is RESOLVING (0) >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Verifying replica's sequence number vs all sequence numbers... >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""] >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Line does not match expected syntax. Ignoring. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61924838473739"] >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61924838473739"] >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 2 sequence numbers were found >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Max sequence number is 3852:B (61924838473739) >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Sequence number of server2 is 3852:B (61924838473739) >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Promoting replica to PRIMARY role... >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Replica is PRIMARY (1) >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Replica is now PRIMARY >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Monitor Caller is: promote. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Replica is PRIMARY (1) >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Instance name is server2. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Current master is server2. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 DB_FAILOVER is ON. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: 2023/07/19 14:56:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: promote: PROMOTION_SCORE: 20 >Jul 19 14:56:14 ag(ag_cluster)[71103]: INFO: ag_cluster promote : 0 >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: mssql_validate >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: Resource agent invoked with: notify >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: mssql_notify post-promote >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: notify: 2023/07/19 14:56:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote] >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: notify: 2023/07/19 14:56:14 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: notify: 2023/07/19 14:56:14 Replica is PRIMARY (1) >Jul 19 14:56:14 ag(ag_cluster)[71374]: INFO: ag_cluster notify : 0 >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: mssql_validate >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: Resource agent invoked with: monitor >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: mssql_monitor >Jul 19 14:56:14 IPaddr2(virtualip)[71461]: INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0 >Jul 19 14:56:14 IPaddr2(virtualip)[71461]: INFO: Bringing device eth0 up >Jul 19 14:56:14 IPaddr2(virtualip)[71461]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:14 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:14 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:56:14 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:14 Connected to the instance at localhost:1433 >Jul 19 14:56:18 IPaddr2(virtualip)[71461]: INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0 >Sent 5 probes (5 broadcast(s)) >Received 0 response(s) >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 Monitor Caller is: monitor. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 Replica is PRIMARY (1) >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 Instance name is server2. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 Current master is server2. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 DB_FAILOVER is ON. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: 2023/07/19 14:56:19 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: lease_expiry after monitor update: >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:56:19 ag(ag_cluster)[71449]: INFO: ag_cluster monitor : 8 >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: mssql_validate >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: Resource agent invoked with: notify >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: mssql_notify pre-stop >Jul 19 14:56:27 ag(ag_cluster)[71801]: INFO: ag_cluster notify : 0 >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: mssql_validate >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: Resource agent invoked with: notify >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: mssql_notify pre-stop >Jul 19 14:56:27 ag(ag_cluster)[71846]: INFO: ag_cluster notify : 0 >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: mssql_validate >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: Resource agent invoked with: notify >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: mssql_notify post-stop >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: notify: 2023/07/19 14:56:27 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop] >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: notify: 2023/07/19 14:56:27 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: notify: 2023/07/19 14:56:27 Replica is PRIMARY (1) >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: notify: 2023/07/19 14:56:27 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: notify: 2023/07/19 14:56:27 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:27 ag(ag_cluster)[71891]: INFO: ag_cluster notify : 0 >Jul 19 14:56:27 ag(ag_cluster)[71978]: INFO: mssql_validate >Jul 19 14:56:27 ag(ag_cluster)[71978]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:27 ag(ag_cluster)[71978]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:27 ag(ag_cluster)[71978]: INFO: Resource agent invoked with: notify >Jul 19 14:56:27 ag(ag_cluster)[71978]: INFO: mssql_notify pre-start >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 Replica is PRIMARY (1) >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: 2023/07/19 14:56:28 Sequence number is 3853:8 (61929133441032) >Jul 19 14:56:28 ag(ag_cluster)[71978]: INFO: notify: SEQUENCE_NUMBER: 61929133441032 >Jul 19 14:56:33 ag(ag_cluster)[71978]: INFO: ag_cluster notify : 0 >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: mssql_validate >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: Resource agent invoked with: monitor >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: mssql_monitor >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:33 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:56:33 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:33 Connected to the instance at localhost:1433 >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 Monitor Caller is: monitor. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 Replica is PRIMARY (1) >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 Instance name is server2. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 Current master is server2. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 DB_FAILOVER is ON. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: 2023/07/19 14:56:38 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: monitor: PROMOTION_SCORE: 20 >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: lease_expiry after monitor update: >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:56:38 ag(ag_cluster)[72089]: INFO: ag_cluster monitor : 8 >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: mssql_validate >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: Resource agent invoked with: notify >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: mssql_notify pre-start >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 Replica is PRIMARY (1) >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: 2023/07/19 14:56:38 Sequence number is 3853:B (61929133441035) >Jul 19 14:56:38 ag(ag_cluster)[72316]: INFO: notify: SEQUENCE_NUMBER: 61929133441035 >Jul 19 14:56:43 ag(ag_cluster)[72316]: INFO: ag_cluster notify : 0 >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: mssql_validate >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: Resource agent invoked with: notify >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: mssql_notify pre-start >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start] >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1] >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 Replica is PRIMARY (1) >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 AG has 3 SYNCHRONOUS_COMMIT replicas. >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1... >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 Replica is SYNCHRONOUS_COMMIT (1). >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: 2023/07/19 14:56:43 Sequence number is 3853:B (61929133441035) >Jul 19 14:56:43 ag(ag_cluster)[72426]: INFO: notify: SEQUENCE_NUMBER: 61929133441035 >Jul 19 14:56:48 ag(ag_cluster)[72426]: INFO: ag_cluster notify : 0 >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: mssql_validate >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: OCF_RESKEY_CRM_meta_interval value: 11000 >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: Resource agent invoked with: monitor >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: mssql_monitor >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor] >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1 >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60] >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:49 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433 >Jul 19 14:56:49 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:49 Connected to the instance at localhost:1433 >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Monitor Caller is: monitor. >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Replica is PRIMARY (1) >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Offlining replica... >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Replica is RESOLVING (0) >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Instance name is server2. >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Current master is server1 >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: server2. >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Setting the role to Secondary. >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Setting replica to SECONDARY role... >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: 2023/07/19 14:56:54 Could not set replica to SECONDARY role. Failover Failed. >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: monitor: PROMOTION_SCORE: -INFINITY >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: lease_expiry after monitor update: >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value="" >name="ag_cluster-lease-expiry" host="server3" value="" >name="ag_cluster-lease-expiry" host="server2" value="" >Jul 19 14:56:54 ag(ag_cluster)[72588]: INFO: ag_cluster monitor : 0 >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: mssql_validate >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: Resource agent invoked with: notify >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: mssql_notify post-start >Jul 19 14:56:54 ag(ag_cluster)[72827]: INFO: ag_cluster notify : 0 >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: mssql_validate >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: Resource agent invoked with: notify >Jul 19 14:56:55 IPaddr2(virtualip)[72873]: INFO: IP status = ok, IP_CIP= >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: mssql_notify pre-demote >Jul 19 14:56:55 ag(ag_cluster)[72874]: INFO: ag_cluster notify : 0 >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: mssql_validate >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: Resource agent invoked with: demote >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: mssql_demote >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: demote: 2023/07/19 14:56:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote] >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: demote: 2023/07/19 14:56:55 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: demote: 2023/07/19 14:56:55 Replica is RESOLVING (0) >Jul 19 14:56:55 ag(ag_cluster)[72972]: INFO: ag_cluster demote : 0 >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: mssql_validate >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: Resource agent invoked with: notify >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: mssql_notify post-demote >Jul 19 14:56:55 ag(ag_cluster)[73064]: INFO: ag_cluster notify : 0 >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: mssql_validate >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: OCF_RESKEY_CRM_meta_timeout value: 60000 >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: Resource agent invoked with: notify >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: mssql_notify pre-stop >Jul 19 14:56:55 ag(ag_cluster)[73109]: INFO: ag_cluster notify : 0 >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: mssql_validate >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: OCF_RESKEY_CRM_meta_interval value: 0 >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: OCF_RESKEY_CRM_meta_timeout value: 10000 >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: Resource agent invoked with: stop >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: mssql_stop >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: stop: 2023/07/19 14:56:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop] >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: stop: 2023/07/19 14:56:55 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2 >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: stop: 2023/07/19 14:56:55 Replica is RESOLVING (0) >Jul 19 14:56:55 ag(ag_cluster)[73154]: INFO: ag_cluster stop : 0
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 2221772
:
1976118
|
1976119
|
1976120
|
1976684
| 1976685 |
1976686