Attachment 1976685 Details for Bug 2221772 – pacemaker logs_updated

pacemaker logs_updated

pacemaker2_7_19.log (text/plain), 440.86 KB, created by Aravind Mahadevan on 2023-07-20 09:09:02 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Aravind Mahadevan

Created: 2023-07-20 09:09:02 UTC

Size: 440.86 KB

patch

obsolete

>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: 2023/07/19 14:37:56 Monitor Caller is: monitor.
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: 2023/07/19 14:37:56 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: 2023/07/19 14:37:56 Replica is SECONDARY (2)
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: 2023/07/19 14:37:56 Instance name is server2.
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: 2023/07/19 14:37:56 Current master is server1.
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: monitor: PROMOTION_SCORE: 10
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:37:56  ag(ag_cluster)[38997]:    INFO: ag_cluster monitor : 0
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: mssql_validate
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: OCF_RESKEY_CRM_meta_interval value: 12000
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: mssql_monitor
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:03 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:03 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60]
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:03 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:03  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:03 Connected to the instance at localhost:1433
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:08 Monitor Caller is: monitor.
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:08 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:08 Replica is SECONDARY (2)
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:08 Instance name is server2.
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: 2023/07/19 14:38:08 Current master is server1.
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: monitor: PROMOTION_SCORE: 10
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:38:08  ag(ag_cluster)[39152]:    INFO: ag_cluster monitor : 0
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: mssql_validate
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: OCF_RESKEY_CRM_meta_interval value: 10000
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: mssql_monitor
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:08 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:08 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60]
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:08 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:08  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:08 Connected to the instance at localhost:1433
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:13 Monitor Caller is: monitor.
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:13 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:13 Replica is SECONDARY (2)
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:13 Instance name is server2.
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: 2023/07/19 14:38:13 Current master is server1.
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: monitor: PROMOTION_SCORE: 10
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:38:13  ag(ag_cluster)[39306]:    INFO: ag_cluster monitor : 0
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: mssql_validate
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: OCF_RESKEY_CRM_meta_interval value: 12000
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: mssql_monitor
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [74]; monitor-interval-timeout [60]
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:20 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:20  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:20 Connected to the instance at localhost:1433
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:25 Monitor Caller is: monitor.
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:25 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:25 Replica is SECONDARY (2)
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:25 Instance name is server2.
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: 2023/07/19 14:38:25 Current master is server1.
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: monitor: PROMOTION_SCORE: 10
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:38:25  ag(ag_cluster)[39460]:    INFO: ag_cluster monitor : 0
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: mssql_validate
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: mssql_notify pre-demote
>Jul 19 14:38:25  ag(ag_cluster)[39614]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: mssql_validate
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: OCF_RESKEY_CRM_meta_interval value: 10000
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: mssql_monitor
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:25 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [72]; monitor-interval-timeout [60]
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:25 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:25  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:25 Connected to the instance at localhost:1433
>Jul 19 14:38:30  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:30 Monitor Caller is: monitor.
>Jul 19 14:38:30  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:30  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:30 Replica is SECONDARY (2)
>Jul 19 14:38:30  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:30 Instance name is server2.
>Jul 19 14:38:31  ag(ag_cluster)[39659]:    INFO: monitor: 2023/07/19 14:38:30 Current master is server1.
>Jul 19 14:38:31  ag(ag_cluster)[39659]:    INFO: monitor: PROMOTION_SCORE: 10
>Jul 19 14:38:31  ag(ag_cluster)[39659]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:38:31  ag(ag_cluster)[39659]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:38:31  ag(ag_cluster)[39659]:    INFO: ag_cluster monitor : 0
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: mssql_validate
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: mssql_notify post-demote
>Jul 19 14:38:31  ag(ag_cluster)[39813]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: mssql_validate
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: mssql_notify pre-stop
>Jul 19 14:38:31  ag(ag_cluster)[39858]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: mssql_validate
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: mssql_notify post-stop
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: notify: 2023/07/19 14:38:31 Replica is SECONDARY (2)
>Jul 19 14:38:31  ag(ag_cluster)[39903]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: mssql_validate
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: mssql_notify pre-start
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: 2023/07/19 14:38:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: 2023/07/19 14:38:31 Replica is SECONDARY (2)
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: 2023/07/19 14:38:31 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: 2023/07/19 14:38:31 Sequence number is 3847:F (61877593833487)
>Jul 19 14:38:31  ag(ag_cluster)[39977]:    INFO: notify: SEQUENCE_NUMBER: 61877593833487
>Jul 19 14:38:36  ag(ag_cluster)[39977]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: mssql_validate
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: mssql_notify pre-stop
>Jul 19 14:38:36  ag(ag_cluster)[40076]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:36  ag(ag_cluster)[40121]:    INFO: mssql_validate
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: Resource agent invoked with: stop
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: mssql_stop
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: stop: 2023/07/19 14:38:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: stop: 2023/07/19 14:38:37 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: stop: 2023/07/19 14:38:37 Replica is SECONDARY (2)
>Jul 19 14:38:37  ag(ag_cluster)[40121]:    INFO: ag_cluster stop : 0
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: mssql_validate
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: Resource agent invoked with: start
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: mssql_start
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:42 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:42 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:42  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:42 Connected to the instance at localhost:1433
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Replica is SECONDARY (2)
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Sequence number is 3847:F (61877593833487)
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Line does not match expected syntax. Ignoring.
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61877593833487"]
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61877593833487"]
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 2 sequence numbers were found
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Max sequence number is 3847:F (61877593833487)
>Jul 19 14:38:47  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Monitor Caller is: start.
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Replica is SECONDARY (2)
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Instance name is server2.
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: start: 2023/07/19 14:38:47 Current master is .
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:38:48  ag(ag_cluster)[40202]:    INFO: ag_cluster start : 0
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: mssql_validate
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: mssql_notify post-start
>Jul 19 14:38:48  ag(ag_cluster)[40402]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: mssql_validate
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: mssql_notify pre-demote
>Jul 19 14:38:53  ag(ag_cluster)[40448]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: mssql_validate
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: mssql_notify post-demote
>Jul 19 14:38:53  ag(ag_cluster)[40493]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: mssql_validate
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: mssql_notify pre-stop
>Jul 19 14:38:53  ag(ag_cluster)[40538]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: mssql_validate
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: mssql_notify post-stop
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: notify: 2023/07/19 14:38:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: notify: 2023/07/19 14:38:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: notify: 2023/07/19 14:38:53 Replica is SECONDARY (2)
>Jul 19 14:38:53  ag(ag_cluster)[40583]:    INFO: ag_cluster notify : 0
>Jul 19 14:38:53  ag(ag_cluster)[40658]:    INFO: mssql_validate
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: Resource agent invoked with: notify
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: mssql_notify pre-promote
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: notify: 2023/07/19 14:38:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: notify: 2023/07/19 14:38:54 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:38:54  ag(ag_cluster)[40658]:    INFO: notify: 2023/07/19 14:38:54 Connected to the instance at localhost:1433
>Jul 19 14:38:59  ag(ag_cluster)[40658]:    INFO: notify: 2023/07/19 14:38:59 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:38:59  ag(ag_cluster)[40658]:    INFO: notify: 2023/07/19 14:38:59 Sequence number is 3847:F (61877593833487)
>Jul 19 14:38:59  ag(ag_cluster)[40658]:    INFO: notify: SEQUENCE_NUMBER: 61877593833487
>Jul 19 14:39:04  ag(ag_cluster)[40658]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: mssql_validate
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: mssql_notify pre-promote
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: notify: 2023/07/19 14:39:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: notify: 2023/07/19 14:39:04 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:39:04  ag(ag_cluster)[40761]:    INFO: notify: 2023/07/19 14:39:04 Connected to the instance at localhost:1433
>Jul 19 14:39:09  ag(ag_cluster)[40761]:    INFO: notify: 2023/07/19 14:39:09 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:39:09  ag(ag_cluster)[40761]:    INFO: notify: 2023/07/19 14:39:09 Sequence number is 3847:F (61877593833487)
>Jul 19 14:39:09  ag(ag_cluster)[40761]:    INFO: notify: SEQUENCE_NUMBER: 61877593833487
>Jul 19 14:39:14  ag(ag_cluster)[40761]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: mssql_validate
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: Resource agent invoked with: promote
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: mssql_promote
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:25 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:25 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:39:25  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:25 Connected to the instance at localhost:1433
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Replica is SECONDARY (2)
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Line does not match expected syntax. Ignoring.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61877593833487"]
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61877593833487"]
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 2 sequence numbers were found
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Max sequence number is 3847:F (61877593833487)
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Sequence number of server2 is 3847:F (61877593833487)
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Promoting replica to PRIMARY role...
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Replica is PRIMARY (1)
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Replica is now PRIMARY
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Monitor Caller is: promote.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Replica is PRIMARY (1)
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Instance name is server2.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Current master is server2.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 DB_FAILOVER is ON.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: 2023/07/19 14:39:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:39:30  ag(ag_cluster)[40885]:    INFO: ag_cluster promote : 0
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: mssql_validate
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: mssql_notify post-promote
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: notify: 2023/07/19 14:39:30 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: notify: 2023/07/19 14:39:30 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: notify: 2023/07/19 14:39:30 Replica is PRIMARY (1)
>Jul 19 14:39:30  ag(ag_cluster)[41161]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: mssql_validate
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: mssql_monitor
>Jul 19 14:39:36  IPaddr2(virtualip)[41244]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:39:36  IPaddr2(virtualip)[41244]:    INFO: Bringing device eth0 up
>Jul 19 14:39:36  IPaddr2(virtualip)[41244]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:36 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:36 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:39:36  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:36 Connected to the instance at localhost:1433
>Jul 19 14:39:40  IPaddr2(virtualip)[41244]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 Monitor Caller is: monitor.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 Replica is PRIMARY (1)
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 Instance name is server2.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 Current master is server2.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 DB_FAILOVER is ON.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: 2023/07/19 14:39:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:39:41  ag(ag_cluster)[41237]:    INFO: ag_cluster monitor : 8
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: mssql_validate
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: mssql_notify pre-stop
>Jul 19 14:39:43  ag(ag_cluster)[41547]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: mssql_validate
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: mssql_notify pre-stop
>Jul 19 14:39:49  ag(ag_cluster)[41642]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: mssql_validate
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: mssql_monitor
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:39:52  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:52 Connected to the instance at localhost:1433
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 Monitor Caller is: monitor.
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 Replica is PRIMARY (1)
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 Instance name is server2.
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 Current master is server2.
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:39:57  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 DB_FAILOVER is ON.
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: monitor: 2023/07/19 14:39:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:39:58  ag(ag_cluster)[41688]:    INFO: ag_cluster monitor : 8
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: mssql_validate
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: mssql_notify pre-stop
>Jul 19 14:39:58  ag(ag_cluster)[41913]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: mssql_validate
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: mssql_notify post-stop
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: notify: 2023/07/19 14:39:58 Replica is PRIMARY (1)
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: notify: 2023/07/19 14:39:58 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: notify: 2023/07/19 14:39:58 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:58  ag(ag_cluster)[41958]:    INFO: ag_cluster notify : 0
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: mssql_validate
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: Resource agent invoked with: notify
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: mssql_notify pre-start
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:39:58  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 Replica is PRIMARY (1)
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:58 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: 2023/07/19 14:39:59 Sequence number is 3848:B (61881888800779)
>Jul 19 14:39:59  ag(ag_cluster)[42045]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:40:04  ag(ag_cluster)[42045]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: mssql_validate
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: mssql_notify pre-start
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 Replica is PRIMARY (1)
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: 2023/07/19 14:40:04 Sequence number is 3848:B (61881888800779)
>Jul 19 14:40:04  ag(ag_cluster)[42155]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:40:09  ag(ag_cluster)[42155]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: mssql_validate
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: mssql_monitor
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:09 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:40:09  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:09 Connected to the instance at localhost:1433
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 Monitor Caller is: monitor.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 Replica is PRIMARY (1)
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 Instance name is server2.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 Current master is server2.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 DB_FAILOVER is ON.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: 2023/07/19 14:40:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:40:14  ag(ag_cluster)[42309]:    INFO: ag_cluster monitor : 8
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: mssql_validate
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: mssql_notify post-start
>Jul 19 14:40:15  ag(ag_cluster)[42493]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: mssql_validate
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: mssql_notify pre-demote
>Jul 19 14:40:15  ag(ag_cluster)[42539]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: mssql_validate
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: mssql_notify post-demote
>Jul 19 14:40:15  ag(ag_cluster)[42584]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: mssql_validate
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: mssql_notify pre-stop
>Jul 19 14:40:15  ag(ag_cluster)[42629]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: mssql_validate
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: Resource agent invoked with: notify
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: mssql_notify post-stop
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: notify: 2023/07/19 14:40:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: notify: 2023/07/19 14:40:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: notify: 2023/07/19 14:40:15 Replica is PRIMARY (1)
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: notify: 2023/07/19 14:40:15 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: notify: 2023/07/19 14:40:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:40:15  ag(ag_cluster)[42674]:    INFO: ag_cluster notify : 0
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: mssql_validate
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: mssql_monitor
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:40:26  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:26 Connected to the instance at localhost:1433
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 Monitor Caller is: monitor.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 Replica is PRIMARY (1)
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 Instance name is server2.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 Current master is server2.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 DB_FAILOVER is ON.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: 2023/07/19 14:40:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:40:31  ag(ag_cluster)[42806]:    INFO: ag_cluster monitor : 8
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: mssql_validate
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: mssql_monitor
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:42 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:42 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:40:42  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:42 Connected to the instance at localhost:1433
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 Monitor Caller is: monitor.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 Replica is PRIMARY (1)
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 Instance name is server2.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 Current master is server2.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 DB_FAILOVER is ON.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: 2023/07/19 14:40:47 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:40:47  ag(ag_cluster)[43074]:    INFO: ag_cluster monitor : 8
>Jul 19 14:40:58  ag(ag_cluster)[43350]:    INFO: mssql_validate
>Jul 19 14:40:58  ag(ag_cluster)[43350]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:40:58  ag(ag_cluster)[43350]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:40:58  ag(ag_cluster)[43350]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:40:58  ag(ag_cluster)[43350]:    INFO: mssql_monitor
>Jul 19 14:40:59  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:40:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:40:59  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:40:59 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:40:59  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:40:59 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:40:59  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:40:59 Connected to the instance at localhost:1433
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 Monitor Caller is: monitor.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 Replica is PRIMARY (1)
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 Instance name is server2.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 Current master is server2.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 DB_FAILOVER is ON.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: 2023/07/19 14:41:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:41:04  ag(ag_cluster)[43350]:    INFO: ag_cluster monitor : 8
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: mssql_validate
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: mssql_notify pre-stop
>Jul 19 14:41:10  ag(ag_cluster)[43581]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: mssql_validate
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: mssql_notify pre-stop
>Jul 19 14:41:13  ag(ag_cluster)[43628]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: mssql_validate
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: mssql_monitor
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:15 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:41:15  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:15 Connected to the instance at localhost:1433
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 Monitor Caller is: monitor.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 Replica is PRIMARY (1)
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 Instance name is server2.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 Current master is server2.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 DB_FAILOVER is ON.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: 2023/07/19 14:41:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:41:20  ag(ag_cluster)[43674]:    INFO: ag_cluster monitor : 8
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: mssql_validate
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: mssql_notify post-stop
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: notify: 2023/07/19 14:41:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: notify: 2023/07/19 14:41:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: notify: 2023/07/19 14:41:20 Replica is PRIMARY (1)
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: notify: 2023/07/19 14:41:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:41:20  ag(ag_cluster)[43902]:    INFO: notify: 2023/07/19 14:41:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:41:21  ag(ag_cluster)[43902]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: mssql_validate
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: mssql_notify pre-start
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 Replica is PRIMARY (1)
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: 2023/07/19 14:41:21 Sequence number is 3848:B (61881888800779)
>Jul 19 14:41:21  ag(ag_cluster)[43989]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:41:26  ag(ag_cluster)[43989]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: mssql_validate
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: mssql_notify pre-start
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 Replica is PRIMARY (1)
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: 2023/07/19 14:41:26 Sequence number is 3848:B (61881888800779)
>Jul 19 14:41:26  ag(ag_cluster)[44101]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:41:31  ag(ag_cluster)[44101]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: mssql_validate
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: mssql_monitor
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:31 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:41:31  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:31 Connected to the instance at localhost:1433
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:36 Monitor Caller is: monitor.
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Replica is PRIMARY (1)
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Offlining replica...
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Replica is RESOLVING (0)
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Instance name is server2.
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Current master is server1
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: server2.
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Setting the role to Secondary.
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Setting replica to SECONDARY role...
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: 2023/07/19 14:41:37 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:41:37  ag(ag_cluster)[44254]:    INFO: ag_cluster monitor : 0
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: mssql_validate
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: mssql_notify post-start
>Jul 19 14:41:37  ag(ag_cluster)[44473]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: mssql_validate
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: mssql_notify pre-demote
>Jul 19 14:41:37  IPaddr2(virtualip)[44538]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:41:37  ag(ag_cluster)[44539]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: mssql_validate
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: Resource agent invoked with: demote
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: mssql_demote
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: demote: 2023/07/19 14:41:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: demote: 2023/07/19 14:41:37 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: demote: 2023/07/19 14:41:37 Replica is RESOLVING (0)
>Jul 19 14:41:37  ag(ag_cluster)[44637]:    INFO: ag_cluster demote : 0
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: mssql_validate
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: mssql_notify post-demote
>Jul 19 14:41:37  ag(ag_cluster)[44730]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:37  ag(ag_cluster)[44775]:    INFO: mssql_validate
>Jul 19 14:41:37  ag(ag_cluster)[44775]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:37  ag(ag_cluster)[44775]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:38  ag(ag_cluster)[44775]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:38  ag(ag_cluster)[44775]:    INFO: mssql_notify pre-stop
>Jul 19 14:41:38  ag(ag_cluster)[44775]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: mssql_validate
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: Resource agent invoked with: stop
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: mssql_stop
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: stop: 2023/07/19 14:41:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: stop: 2023/07/19 14:41:38 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: stop: 2023/07/19 14:41:38 Replica is RESOLVING (0)
>Jul 19 14:41:38  ag(ag_cluster)[44820]:    INFO: ag_cluster stop : 0
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: mssql_validate
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: Resource agent invoked with: start
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: mssql_start
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:48 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:41:48  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:48 Connected to the instance at localhost:1433
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Replica is RESOLVING (0)
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Sequence number is 3848:B (61881888800779)
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Line does not match expected syntax. Ignoring.
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61881888800779"]
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:41:53  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Line does not match expected syntax. Ignoring.
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 1 sequence numbers were found
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Max sequence number is 3848:B (61881888800779)
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Monitor Caller is: start.
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Replica is RESOLVING (0)
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Instance name is server2.
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: 2023/07/19 14:41:53 Current master is .
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:41:54  ag(ag_cluster)[44914]:    INFO: ag_cluster start : 0
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: mssql_validate
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: mssql_notify post-start
>Jul 19 14:41:54  ag(ag_cluster)[45112]:    INFO: ag_cluster notify : 0
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: mssql_validate
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: Resource agent invoked with: notify
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: mssql_notify pre-promote
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: notify: 2023/07/19 14:41:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: notify: 2023/07/19 14:41:54 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:41:54  ag(ag_cluster)[45158]:    INFO: notify: 2023/07/19 14:41:54 Connected to the instance at localhost:1433
>Jul 19 14:41:59  ag(ag_cluster)[45158]:    INFO: notify: 2023/07/19 14:41:59 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:41:59  ag(ag_cluster)[45158]:    INFO: notify: 2023/07/19 14:41:59 Sequence number is 3848:B (61881888800779)
>Jul 19 14:41:59  ag(ag_cluster)[45158]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:42:04  ag(ag_cluster)[45158]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: mssql_validate
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: mssql_notify pre-promote
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: notify: 2023/07/19 14:42:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: notify: 2023/07/19 14:42:04 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:42:04  ag(ag_cluster)[45266]:    INFO: notify: 2023/07/19 14:42:04 Connected to the instance at localhost:1433
>Jul 19 14:42:09  ag(ag_cluster)[45266]:    INFO: notify: 2023/07/19 14:42:09 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:42:09  ag(ag_cluster)[45266]:    INFO: notify: 2023/07/19 14:42:09 Sequence number is 3848:B (61881888800779)
>Jul 19 14:42:09  ag(ag_cluster)[45266]:    INFO: notify: SEQUENCE_NUMBER: 61881888800779
>Jul 19 14:42:14  ag(ag_cluster)[45266]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: mssql_validate
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: Resource agent invoked with: promote
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: mssql_promote
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:15 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:15 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:42:15  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:15 Connected to the instance at localhost:1433
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Replica is RESOLVING (0)
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Line does not match expected syntax. Ignoring.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61881888800779"]
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61881888800779"]
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 2 sequence numbers were found
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Max sequence number is 3848:B (61881888800779)
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Sequence number of server2 is 3848:B (61881888800779)
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Promoting replica to PRIMARY role...
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Replica is PRIMARY (1)
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Replica is now PRIMARY
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Monitor Caller is: promote.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Replica is PRIMARY (1)
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Instance name is server2.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Current master is server2.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 DB_FAILOVER is ON.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: 2023/07/19 14:42:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:42:20  ag(ag_cluster)[45371]:    INFO: ag_cluster promote : 0
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: mssql_validate
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: mssql_notify post-promote
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: notify: 2023/07/19 14:42:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: notify: 2023/07/19 14:42:20 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: notify: 2023/07/19 14:42:20 Replica is PRIMARY (1)
>Jul 19 14:42:20  ag(ag_cluster)[45642]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: mssql_validate
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: mssql_monitor
>Jul 19 14:42:20  IPaddr2(virtualip)[45726]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:42:20  IPaddr2(virtualip)[45726]:    INFO: Bringing device eth0 up
>Jul 19 14:42:20  IPaddr2(virtualip)[45726]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:20 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:42:20  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:20 Connected to the instance at localhost:1433
>Jul 19 14:42:24  IPaddr2(virtualip)[45726]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 Monitor Caller is: monitor.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 Replica is PRIMARY (1)
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 Instance name is server2.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 Current master is server2.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 DB_FAILOVER is ON.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: 2023/07/19 14:42:25 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:25  ag(ag_cluster)[45718]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:42:26  ag(ag_cluster)[45718]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:42:26  ag(ag_cluster)[45718]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:42:26  ag(ag_cluster)[45718]:    INFO: ag_cluster monitor : 8
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: mssql_validate
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: mssql_notify pre-stop
>Jul 19 14:42:32  ag(ag_cluster)[46073]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: mssql_validate
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: mssql_notify pre-stop
>Jul 19 14:42:32  ag(ag_cluster)[46118]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: mssql_validate
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: mssql_notify post-stop
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: notify: 2023/07/19 14:42:33 Replica is PRIMARY (1)
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: notify: 2023/07/19 14:42:33 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: notify: 2023/07/19 14:42:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:33  ag(ag_cluster)[46165]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: mssql_validate
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: mssql_notify pre-start
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 Replica is PRIMARY (1)
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: 2023/07/19 14:42:33 Sequence number is 3849:6 (61886183768070)
>Jul 19 14:42:33  ag(ag_cluster)[46253]:    INFO: notify: SEQUENCE_NUMBER: 61886183768070
>Jul 19 14:42:38  ag(ag_cluster)[46253]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: mssql_validate
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: mssql_monitor
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:38 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:42:38  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:38 Connected to the instance at localhost:1433
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 Monitor Caller is: monitor.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 Replica is PRIMARY (1)
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 Instance name is server2.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 Current master is server2.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 DB_FAILOVER is ON.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: 2023/07/19 14:42:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:42:43  ag(ag_cluster)[46365]:    INFO: ag_cluster monitor : 8
>Jul 19 14:42:43  ag(ag_cluster)[46591]:    INFO: mssql_validate
>Jul 19 14:42:43  ag(ag_cluster)[46591]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:43  ag(ag_cluster)[46591]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: mssql_notify pre-start
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 Replica is PRIMARY (1)
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: 2023/07/19 14:42:44 Sequence number is 3849:B (61886183768075)
>Jul 19 14:42:44  ag(ag_cluster)[46591]:    INFO: notify: SEQUENCE_NUMBER: 61886183768075
>Jul 19 14:42:49  ag(ag_cluster)[46591]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: mssql_validate
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: Resource agent invoked with: notify
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: mssql_notify pre-start
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 Replica is PRIMARY (1)
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: 2023/07/19 14:42:49 Sequence number is 3849:B (61886183768075)
>Jul 19 14:42:49  ag(ag_cluster)[46711]:    INFO: notify: SEQUENCE_NUMBER: 61886183768075
>Jul 19 14:42:54  ag(ag_cluster)[46711]:    INFO: ag_cluster notify : 0
>Jul 19 14:42:54  ag(ag_cluster)[46866]:    INFO: mssql_validate
>Jul 19 14:42:54  ag(ag_cluster)[46866]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:42:54  ag(ag_cluster)[46866]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: mssql_monitor
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:42:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:42:55 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:42:55 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:42:55  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:42:55 Connected to the instance at localhost:1433
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Monitor Caller is: monitor.
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Replica is PRIMARY (1)
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Offlining replica...
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Replica is RESOLVING (0)
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Instance name is server2.
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Current master is server1
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: server2.
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Setting the role to Secondary.
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Setting replica to SECONDARY role...
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: 2023/07/19 14:43:00 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:43:00  ag(ag_cluster)[46866]:    INFO: ag_cluster monitor : 0
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: mssql_validate
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: mssql_notify post-start
>Jul 19 14:43:00  ag(ag_cluster)[47063]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: mssql_validate
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: mssql_notify pre-demote
>Jul 19 14:43:00  IPaddr2(virtualip)[47109]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:43:00  ag(ag_cluster)[47110]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:00  ag(ag_cluster)[47208]:    INFO: mssql_validate
>Jul 19 14:43:00  ag(ag_cluster)[47208]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:00  ag(ag_cluster)[47208]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:43:00  ag(ag_cluster)[47208]:    INFO: Resource agent invoked with: demote
>Jul 19 14:43:01  ag(ag_cluster)[47208]:    INFO: mssql_demote
>Jul 19 14:43:01  ag(ag_cluster)[47208]:    INFO: demote: 2023/07/19 14:43:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:43:01  ag(ag_cluster)[47208]:    INFO: demote: 2023/07/19 14:43:01 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:01  ag(ag_cluster)[47208]:    INFO: demote: 2023/07/19 14:43:01 Replica is RESOLVING (0)
>Jul 19 14:43:01  ag(ag_cluster)[47208]:    INFO: ag_cluster demote : 0
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: mssql_validate
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: mssql_notify post-demote
>Jul 19 14:43:01  ag(ag_cluster)[47301]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: mssql_validate
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: mssql_notify pre-stop
>Jul 19 14:43:01  ag(ag_cluster)[47346]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: mssql_validate
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: Resource agent invoked with: stop
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: mssql_stop
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: stop: 2023/07/19 14:43:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: stop: 2023/07/19 14:43:01 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: stop: 2023/07/19 14:43:01 Replica is RESOLVING (0)
>Jul 19 14:43:01  ag(ag_cluster)[47391]:    INFO: ag_cluster stop : 0
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: mssql_validate
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: Resource agent invoked with: start
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: mssql_start
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:12 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:12 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:43:12  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:12 Connected to the instance at localhost:1433
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Replica is RESOLVING (0)
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Sequence number is 3849:B (61886183768075)
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Line does not match expected syntax. Ignoring.
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61886183768075"]
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Line does not match expected syntax. Ignoring.
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 1 sequence numbers were found
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Max sequence number is 3849:B (61886183768075)
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Monitor Caller is: start.
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Replica is RESOLVING (0)
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Instance name is server2.
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: 2023/07/19 14:43:17 Current master is .
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:43:17  ag(ag_cluster)[47474]:    INFO: ag_cluster start : 0
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: mssql_validate
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: mssql_notify post-start
>Jul 19 14:43:17  ag(ag_cluster)[47671]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: mssql_validate
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: mssql_notify pre-promote
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: notify: 2023/07/19 14:43:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: notify: 2023/07/19 14:43:17 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:43:17  ag(ag_cluster)[47717]:    INFO: notify: 2023/07/19 14:43:17 Connected to the instance at localhost:1433
>Jul 19 14:43:22  ag(ag_cluster)[47717]:    INFO: notify: 2023/07/19 14:43:22 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:43:22  ag(ag_cluster)[47717]:    INFO: notify: 2023/07/19 14:43:22 Sequence number is 3849:B (61886183768075)
>Jul 19 14:43:22  ag(ag_cluster)[47717]:    INFO: notify: SEQUENCE_NUMBER: 61886183768075
>Jul 19 14:43:27  ag(ag_cluster)[47717]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: mssql_validate
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: mssql_notify pre-promote
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: notify: 2023/07/19 14:43:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: notify: 2023/07/19 14:43:28 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:43:28  ag(ag_cluster)[47823]:    INFO: notify: 2023/07/19 14:43:28 Connected to the instance at localhost:1433
>Jul 19 14:43:33  ag(ag_cluster)[47823]:    INFO: notify: 2023/07/19 14:43:33 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:43:33  ag(ag_cluster)[47823]:    INFO: notify: 2023/07/19 14:43:33 Sequence number is 3849:B (61886183768075)
>Jul 19 14:43:33  ag(ag_cluster)[47823]:    INFO: notify: SEQUENCE_NUMBER: 61886183768075
>Jul 19 14:43:38  ag(ag_cluster)[47823]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: mssql_validate
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: Resource agent invoked with: promote
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: mssql_promote
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:38 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:38 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:43:38  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:38 Connected to the instance at localhost:1433
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Replica is RESOLVING (0)
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Line does not match expected syntax. Ignoring.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61886183768075"]
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61886183768075"]
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 2 sequence numbers were found
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Max sequence number is 3849:B (61886183768075)
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Sequence number of server2 is 3849:B (61886183768075)
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Promoting replica to PRIMARY role...
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Replica is PRIMARY (1)
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Replica is now PRIMARY
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Monitor Caller is: promote.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Replica is PRIMARY (1)
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Instance name is server2.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Current master is server2.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 DB_FAILOVER is ON.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: 2023/07/19 14:43:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:43:43  ag(ag_cluster)[47930]:    INFO: ag_cluster promote : 0
>Jul 19 14:43:43  ag(ag_cluster)[48204]:    INFO: mssql_validate
>Jul 19 14:43:43  ag(ag_cluster)[48204]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:43  ag(ag_cluster)[48204]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:43  ag(ag_cluster)[48204]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:43  ag(ag_cluster)[48204]:    INFO: mssql_notify post-promote
>Jul 19 14:43:44  ag(ag_cluster)[48204]:    INFO: notify: 2023/07/19 14:43:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:43:44  ag(ag_cluster)[48204]:    INFO: notify: 2023/07/19 14:43:44 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:44  ag(ag_cluster)[48204]:    INFO: notify: 2023/07/19 14:43:44 Replica is PRIMARY (1)
>Jul 19 14:43:44  ag(ag_cluster)[48204]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: mssql_validate
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: mssql_monitor
>Jul 19 14:43:44  IPaddr2(virtualip)[48292]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:43:44  IPaddr2(virtualip)[48292]:    INFO: Bringing device eth0 up
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:44 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:43:44  IPaddr2(virtualip)[48292]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:44 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:44 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:43:44  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:44 Connected to the instance at localhost:1433
>Jul 19 14:43:48  IPaddr2(virtualip)[48292]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 Monitor Caller is: monitor.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 Replica is PRIMARY (1)
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 Instance name is server2.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 Current master is server2.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 DB_FAILOVER is ON.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: 2023/07/19 14:43:49 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:43:49  ag(ag_cluster)[48280]:    INFO: ag_cluster monitor : 8
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: mssql_validate
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: mssql_notify pre-stop
>Jul 19 14:43:55  ag(ag_cluster)[48646]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: mssql_validate
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: mssql_notify pre-stop
>Jul 19 14:43:55  ag(ag_cluster)[48691]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: mssql_validate
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: mssql_notify post-stop
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: notify: 2023/07/19 14:43:56 Replica is PRIMARY (1)
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: notify: 2023/07/19 14:43:56 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: notify: 2023/07/19 14:43:56 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:43:56  ag(ag_cluster)[48736]:    INFO: ag_cluster notify : 0
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: mssql_validate
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: Resource agent invoked with: notify
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: mssql_notify pre-start
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 Replica is PRIMARY (1)
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: 2023/07/19 14:43:56 Sequence number is 384A:6 (61890478735366)
>Jul 19 14:43:56  ag(ag_cluster)[48822]:    INFO: notify: SEQUENCE_NUMBER: 61890478735366
>Jul 19 14:44:01  ag(ag_cluster)[48822]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: mssql_validate
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: mssql_monitor
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:01 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:01 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:44:01  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:01 Connected to the instance at localhost:1433
>Jul 19 14:44:06  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 Monitor Caller is: monitor.
>Jul 19 14:44:06  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:06  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 Replica is PRIMARY (1)
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 Instance name is server2.
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 Current master is server2.
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 DB_FAILOVER is ON.
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: 2023/07/19 14:44:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:44:07  ag(ag_cluster)[48936]:    INFO: ag_cluster monitor : 8
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: mssql_validate
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: mssql_notify pre-start
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 Replica is PRIMARY (1)
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: 2023/07/19 14:44:07 Sequence number is 384A:B (61890478735371)
>Jul 19 14:44:07  ag(ag_cluster)[49164]:    INFO: notify: SEQUENCE_NUMBER: 61890478735371
>Jul 19 14:44:12  ag(ag_cluster)[49164]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: mssql_validate
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: mssql_notify pre-start
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 Replica is PRIMARY (1)
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: 2023/07/19 14:44:12 Sequence number is 384A:B (61890478735371)
>Jul 19 14:44:12  ag(ag_cluster)[49277]:    INFO: notify: SEQUENCE_NUMBER: 61890478735371
>Jul 19 14:44:17  ag(ag_cluster)[49277]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: mssql_validate
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: mssql_monitor
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:18 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:18 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:18 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:44:18  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:18 Connected to the instance at localhost:1433
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Monitor Caller is: monitor.
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Replica is PRIMARY (1)
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Offlining replica...
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Replica is RESOLVING (0)
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Instance name is server2.
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Current master is server1
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: server2.
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Setting the role to Secondary.
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Setting replica to SECONDARY role...
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: 2023/07/19 14:44:23 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:44:23  ag(ag_cluster)[49432]:    INFO: ag_cluster monitor : 0
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: mssql_validate
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: mssql_notify post-start
>Jul 19 14:44:23  ag(ag_cluster)[49635]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: mssql_validate
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:23  IPaddr2(virtualip)[49681]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: mssql_notify pre-demote
>Jul 19 14:44:23  ag(ag_cluster)[49682]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: mssql_validate
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: Resource agent invoked with: demote
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: mssql_demote
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: demote: 2023/07/19 14:44:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: demote: 2023/07/19 14:44:23 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: demote: 2023/07/19 14:44:23 Replica is RESOLVING (0)
>Jul 19 14:44:23  ag(ag_cluster)[49780]:    INFO: ag_cluster demote : 0
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: mssql_validate
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: mssql_notify post-demote
>Jul 19 14:44:23  ag(ag_cluster)[49873]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:23  ag(ag_cluster)[49918]:    INFO: mssql_validate
>Jul 19 14:44:23  ag(ag_cluster)[49918]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:23  ag(ag_cluster)[49918]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:24  ag(ag_cluster)[49918]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:24  ag(ag_cluster)[49918]:    INFO: mssql_notify pre-stop
>Jul 19 14:44:24  ag(ag_cluster)[49918]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: mssql_validate
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: Resource agent invoked with: stop
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: mssql_stop
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: stop: 2023/07/19 14:44:24 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: stop: 2023/07/19 14:44:24 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: stop: 2023/07/19 14:44:24 Replica is RESOLVING (0)
>Jul 19 14:44:24  ag(ag_cluster)[49963]:    INFO: ag_cluster stop : 0
>Jul 19 14:44:34  ag(ag_cluster)[50047]:    INFO: mssql_validate
>Jul 19 14:44:34  ag(ag_cluster)[50047]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:34  ag(ag_cluster)[50047]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:34  ag(ag_cluster)[50047]:    INFO: Resource agent invoked with: start
>Jul 19 14:44:34  ag(ag_cluster)[50047]:    INFO: mssql_start
>Jul 19 14:44:35  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:44:35  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:35 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:44:35  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:35 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:44:35  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:35 Connected to the instance at localhost:1433
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Replica is RESOLVING (0)
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Sequence number is 384A:B (61890478735371)
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Line does not match expected syntax. Ignoring.
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61890478735371"]
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Line does not match expected syntax. Ignoring.
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 1 sequence numbers were found
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Max sequence number is 384A:B (61890478735371)
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Monitor Caller is: start.
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Replica is RESOLVING (0)
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Instance name is server2.
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: 2023/07/19 14:44:40 Current master is .
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:44:40  ag(ag_cluster)[50047]:    INFO: ag_cluster start : 0
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: mssql_validate
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: mssql_notify post-start
>Jul 19 14:44:40  ag(ag_cluster)[50249]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: mssql_validate
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: mssql_notify pre-promote
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: notify: 2023/07/19 14:44:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: notify: 2023/07/19 14:44:40 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:44:40  ag(ag_cluster)[50295]:    INFO: notify: 2023/07/19 14:44:40 Connected to the instance at localhost:1433
>Jul 19 14:44:45  ag(ag_cluster)[50295]:    INFO: notify: 2023/07/19 14:44:45 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:44:45  ag(ag_cluster)[50295]:    INFO: notify: 2023/07/19 14:44:45 Sequence number is 384A:B (61890478735371)
>Jul 19 14:44:45  ag(ag_cluster)[50295]:    INFO: notify: SEQUENCE_NUMBER: 61890478735371
>Jul 19 14:44:50  ag(ag_cluster)[50295]:    INFO: ag_cluster notify : 0
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: mssql_validate
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: Resource agent invoked with: notify
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: mssql_notify pre-promote
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: notify: 2023/07/19 14:44:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: notify: 2023/07/19 14:44:51 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:44:51  ag(ag_cluster)[50409]:    INFO: notify: 2023/07/19 14:44:51 Connected to the instance at localhost:1433
>Jul 19 14:44:56  ag(ag_cluster)[50409]:    INFO: notify: 2023/07/19 14:44:56 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:44:56  ag(ag_cluster)[50409]:    INFO: notify: 2023/07/19 14:44:56 Sequence number is 384A:B (61890478735371)
>Jul 19 14:44:56  ag(ag_cluster)[50409]:    INFO: notify: SEQUENCE_NUMBER: 61890478735371
>Jul 19 14:45:01  ag(ag_cluster)[50409]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: mssql_validate
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: Resource agent invoked with: promote
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: mssql_promote
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:01 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:01 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:45:01  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:01 Connected to the instance at localhost:1433
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Replica is RESOLVING (0)
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Line does not match expected syntax. Ignoring.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61890478735371"]
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61890478735371"]
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 2 sequence numbers were found
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Max sequence number is 384A:B (61890478735371)
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Sequence number of server2 is 384A:B (61890478735371)
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Promoting replica to PRIMARY role...
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Replica is PRIMARY (1)
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Replica is now PRIMARY
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Monitor Caller is: promote.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Replica is PRIMARY (1)
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Instance name is server2.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Current master is server2.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 DB_FAILOVER is ON.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: 2023/07/19 14:45:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:45:06  ag(ag_cluster)[50514]:    INFO: ag_cluster promote : 0
>Jul 19 14:45:06  ag(ag_cluster)[50788]:    INFO: mssql_validate
>Jul 19 14:45:06  ag(ag_cluster)[50788]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:06  ag(ag_cluster)[50788]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: mssql_notify post-promote
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: notify: 2023/07/19 14:45:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: notify: 2023/07/19 14:45:07 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: notify: 2023/07/19 14:45:07 Replica is PRIMARY (1)
>Jul 19 14:45:07  ag(ag_cluster)[50788]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: mssql_validate
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: mssql_monitor
>Jul 19 14:45:07  IPaddr2(virtualip)[50878]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:45:07  IPaddr2(virtualip)[50878]:    INFO: Bringing device eth0 up
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:45:07  IPaddr2(virtualip)[50878]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:07 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:45:07  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:07 Connected to the instance at localhost:1433
>Jul 19 14:45:11  IPaddr2(virtualip)[50878]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 Monitor Caller is: monitor.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 Replica is PRIMARY (1)
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 Instance name is server2.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 Current master is server2.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 DB_FAILOVER is ON.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: 2023/07/19 14:45:12 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:45:12  ag(ag_cluster)[50864]:    INFO: ag_cluster monitor : 8
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: mssql_validate
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: mssql_notify pre-stop
>Jul 19 14:45:20  ag(ag_cluster)[51220]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: mssql_validate
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: mssql_notify pre-stop
>Jul 19 14:45:20  ag(ag_cluster)[51265]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: mssql_validate
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: mssql_notify post-stop
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: notify: 2023/07/19 14:45:20 Replica is PRIMARY (1)
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: notify: 2023/07/19 14:45:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: notify: 2023/07/19 14:45:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:20  ag(ag_cluster)[51310]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: mssql_validate
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: mssql_notify pre-start
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 Replica is PRIMARY (1)
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: 2023/07/19 14:45:20 Sequence number is 384B:8 (61894773702664)
>Jul 19 14:45:20  ag(ag_cluster)[51397]:    INFO: notify: SEQUENCE_NUMBER: 61894773702664
>Jul 19 14:45:25  ag(ag_cluster)[51397]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: mssql_validate
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: mssql_monitor
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:45:26  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:26 Connected to the instance at localhost:1433
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 Monitor Caller is: monitor.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 Replica is PRIMARY (1)
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 Instance name is server2.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 Current master is server2.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 DB_FAILOVER is ON.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: 2023/07/19 14:45:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:45:31  ag(ag_cluster)[51511]:    INFO: ag_cluster monitor : 8
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: mssql_validate
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: mssql_notify pre-start
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 Replica is PRIMARY (1)
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: 2023/07/19 14:45:31 Sequence number is 384B:B (61894773702667)
>Jul 19 14:45:31  ag(ag_cluster)[51737]:    INFO: notify: SEQUENCE_NUMBER: 61894773702667
>Jul 19 14:45:36  ag(ag_cluster)[51737]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: mssql_validate
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: mssql_notify pre-start
>Jul 19 14:45:36  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:36 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:37 Replica is PRIMARY (1)
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:37 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:37 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:37 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: 2023/07/19 14:45:37 Sequence number is 384B:B (61894773702667)
>Jul 19 14:45:37  ag(ag_cluster)[51849]:    INFO: notify: SEQUENCE_NUMBER: 61894773702667
>Jul 19 14:45:42  ag(ag_cluster)[51849]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: mssql_validate
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: mssql_monitor
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:42 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:42 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:42 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:45:42  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:42 Connected to the instance at localhost:1433
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Monitor Caller is: monitor.
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Replica is PRIMARY (1)
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Offlining replica...
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Replica is RESOLVING (0)
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Instance name is server2.
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Current master is server1
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: server2.
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Setting the role to Secondary.
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Setting replica to SECONDARY role...
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: 2023/07/19 14:45:47 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:45:47  ag(ag_cluster)[52004]:    INFO: ag_cluster monitor : 0
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: mssql_notify post-start
>Jul 19 14:45:48  ag(ag_cluster)[52252]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: mssql_notify pre-demote
>Jul 19 14:45:48  IPaddr2(virtualip)[52298]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:45:48  ag(ag_cluster)[52299]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: Resource agent invoked with: demote
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: mssql_demote
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: demote: 2023/07/19 14:45:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: demote: 2023/07/19 14:45:48 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: demote: 2023/07/19 14:45:48 Replica is RESOLVING (0)
>Jul 19 14:45:48  ag(ag_cluster)[52397]:    INFO: ag_cluster demote : 0
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: mssql_notify post-demote
>Jul 19 14:45:48  ag(ag_cluster)[52490]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: Resource agent invoked with: notify
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: mssql_notify pre-stop
>Jul 19 14:45:48  ag(ag_cluster)[52535]:    INFO: ag_cluster notify : 0
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: mssql_validate
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: Resource agent invoked with: stop
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: mssql_stop
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: stop: 2023/07/19 14:45:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: stop: 2023/07/19 14:45:48 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: stop: 2023/07/19 14:45:48 Replica is RESOLVING (0)
>Jul 19 14:45:48  ag(ag_cluster)[52580]:    INFO: ag_cluster stop : 0
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: mssql_validate
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: Resource agent invoked with: start
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: mssql_start
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:45:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:45:59 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:45:59 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:45:59  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:45:59 Connected to the instance at localhost:1433
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Replica is RESOLVING (0)
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Sequence number is 384B:B (61894773702667)
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Line does not match expected syntax. Ignoring.
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61894773702667"]
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Line does not match expected syntax. Ignoring.
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 1 sequence numbers were found
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Max sequence number is 384B:B (61894773702667)
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Monitor Caller is: start.
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Replica is RESOLVING (0)
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Instance name is server2.
>Jul 19 14:46:04  ag(ag_cluster)[52665]:    INFO: start: 2023/07/19 14:46:04 Current master is .
>Jul 19 14:46:05  ag(ag_cluster)[52665]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:46:05  ag(ag_cluster)[52665]:    INFO: ag_cluster start : 0
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: mssql_validate
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: mssql_notify post-start
>Jul 19 14:46:05  ag(ag_cluster)[52865]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: mssql_validate
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: mssql_notify pre-promote
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: notify: 2023/07/19 14:46:05 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: notify: 2023/07/19 14:46:05 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:46:05  ag(ag_cluster)[52911]:    INFO: notify: 2023/07/19 14:46:05 Connected to the instance at localhost:1433
>Jul 19 14:46:10  ag(ag_cluster)[52911]:    INFO: notify: 2023/07/19 14:46:10 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:46:10  ag(ag_cluster)[52911]:    INFO: notify: 2023/07/19 14:46:10 Sequence number is 384B:B (61894773702667)
>Jul 19 14:46:10  ag(ag_cluster)[52911]:    INFO: notify: SEQUENCE_NUMBER: 61894773702667
>Jul 19 14:46:15  ag(ag_cluster)[52911]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: mssql_validate
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: mssql_notify pre-promote
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: notify: 2023/07/19 14:46:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: notify: 2023/07/19 14:46:15 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:46:15  ag(ag_cluster)[53018]:    INFO: notify: 2023/07/19 14:46:15 Connected to the instance at localhost:1433
>Jul 19 14:46:20  ag(ag_cluster)[53018]:    INFO: notify: 2023/07/19 14:46:20 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:46:20  ag(ag_cluster)[53018]:    INFO: notify: 2023/07/19 14:46:20 Sequence number is 384B:B (61894773702667)
>Jul 19 14:46:20  ag(ag_cluster)[53018]:    INFO: notify: SEQUENCE_NUMBER: 61894773702667
>Jul 19 14:46:25  ag(ag_cluster)[53018]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: mssql_validate
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: Resource agent invoked with: promote
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: mssql_promote
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:25 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:25 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:25 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:46:25  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:25 Connected to the instance at localhost:1433
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Replica is RESOLVING (0)
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Line does not match expected syntax. Ignoring.
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61894773702667"]
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61894773702667"]
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 2 sequence numbers were found
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Max sequence number is 384B:B (61894773702667)
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Sequence number of server2 is 384B:B (61894773702667)
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Promoting replica to PRIMARY role...
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Replica is PRIMARY (1)
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Replica is now PRIMARY
>Jul 19 14:46:30  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Monitor Caller is: promote.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Replica is PRIMARY (1)
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Instance name is server2.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:30 Current master is server2.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:31 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:31 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:31 DB_FAILOVER is ON.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:31 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: 2023/07/19 14:46:31 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:46:31  ag(ag_cluster)[53123]:    INFO: ag_cluster promote : 0
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: mssql_validate
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: mssql_notify post-promote
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: notify: 2023/07/19 14:46:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: notify: 2023/07/19 14:46:31 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: notify: 2023/07/19 14:46:31 Replica is PRIMARY (1)
>Jul 19 14:46:31  ag(ag_cluster)[53400]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: mssql_validate
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: mssql_monitor
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:31 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:46:31  IPaddr2(virtualip)[53487]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:31 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:46:31  IPaddr2(virtualip)[53487]:    INFO: Bringing device eth0 up
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:31 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:46:31  IPaddr2(virtualip)[53487]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:46:31  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:31 Connected to the instance at localhost:1433
>Jul 19 14:46:35  IPaddr2(virtualip)[53487]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 Monitor Caller is: monitor.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 Replica is PRIMARY (1)
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 Instance name is server2.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 Current master is server2.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 DB_FAILOVER is ON.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: 2023/07/19 14:46:36 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:46:36  ag(ag_cluster)[53476]:    INFO: ag_cluster monitor : 8
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: mssql_validate
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: mssql_notify pre-stop
>Jul 19 14:46:43  ag(ag_cluster)[53831]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: mssql_validate
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: mssql_notify pre-stop
>Jul 19 14:46:43  ag(ag_cluster)[53876]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: mssql_validate
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: mssql_notify post-stop
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: notify: 2023/07/19 14:46:43 Replica is PRIMARY (1)
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: notify: 2023/07/19 14:46:43 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: notify: 2023/07/19 14:46:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:43  ag(ag_cluster)[53921]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: mssql_validate
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: mssql_notify pre-start
>Jul 19 14:46:43  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:44 Replica is PRIMARY (1)
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:44 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:44 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:44 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: 2023/07/19 14:46:44 Sequence number is 384C:6 (61899068669958)
>Jul 19 14:46:44  ag(ag_cluster)[54008]:    INFO: notify: SEQUENCE_NUMBER: 61899068669958
>Jul 19 14:46:49  ag(ag_cluster)[54008]:    INFO: ag_cluster notify : 0
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: mssql_validate
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: mssql_monitor
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:49 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:46:49  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:49 Connected to the instance at localhost:1433
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 Monitor Caller is: monitor.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 Replica is PRIMARY (1)
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 Instance name is server2.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 Current master is server2.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 DB_FAILOVER is ON.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: 2023/07/19 14:46:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:46:54  ag(ag_cluster)[54128]:    INFO: ag_cluster monitor : 8
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: mssql_validate
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: Resource agent invoked with: notify
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: mssql_notify pre-start
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 Replica is PRIMARY (1)
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: 2023/07/19 14:46:54 Sequence number is 384C:7 (61899068669959)
>Jul 19 14:46:54  ag(ag_cluster)[54356]:    INFO: notify: SEQUENCE_NUMBER: 61899068669959
>Jul 19 14:47:00  ag(ag_cluster)[54356]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: mssql_validate
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: mssql_notify pre-start
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 Replica is PRIMARY (1)
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: 2023/07/19 14:47:00 Sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:00  ag(ag_cluster)[54469]:    INFO: notify: SEQUENCE_NUMBER: 61899068669959
>Jul 19 14:47:05  ag(ag_cluster)[54469]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: mssql_validate
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: mssql_monitor
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:05 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:05 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:05 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:05  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:05 Connected to the instance at localhost:1433
>Jul 19 14:47:10  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:10 Monitor Caller is: monitor.
>Jul 19 14:47:10  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:10 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:10  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:10 Replica is PRIMARY (1)
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:10 Offlining replica...
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Replica is RESOLVING (0)
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Instance name is server2.
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Current master is server1
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: server2.
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Setting the role to Secondary.
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Setting replica to SECONDARY role...
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: 2023/07/19 14:47:11 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:47:11  ag(ag_cluster)[54624]:    INFO: ag_cluster monitor : 0
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: mssql_notify post-start
>Jul 19 14:47:11  ag(ag_cluster)[54822]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: mssql_notify pre-demote
>Jul 19 14:47:11  IPaddr2(virtualip)[54868]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:47:11  ag(ag_cluster)[54869]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: Resource agent invoked with: demote
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: mssql_demote
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: demote: 2023/07/19 14:47:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: demote: 2023/07/19 14:47:11 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: demote: 2023/07/19 14:47:11 Replica is RESOLVING (0)
>Jul 19 14:47:11  ag(ag_cluster)[54968]:    INFO: ag_cluster demote : 0
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: mssql_notify post-demote
>Jul 19 14:47:11  ag(ag_cluster)[55061]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: mssql_notify pre-stop
>Jul 19 14:47:11  ag(ag_cluster)[55106]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: mssql_validate
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: Resource agent invoked with: stop
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: mssql_stop
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: stop: 2023/07/19 14:47:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: stop: 2023/07/19 14:47:11 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: stop: 2023/07/19 14:47:11 Replica is RESOLVING (0)
>Jul 19 14:47:11  ag(ag_cluster)[55151]:    INFO: ag_cluster stop : 0
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: mssql_validate
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: Resource agent invoked with: start
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: mssql_start
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:22 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:22 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:22 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:22  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:22 Connected to the instance at localhost:1433
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Replica is RESOLVING (0)
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Line does not match expected syntax. Ignoring.
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61899068669959"]
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Line does not match expected syntax. Ignoring.
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 1 sequence numbers were found
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Max sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 Monitor Caller is: start.
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:27 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:28 Replica is RESOLVING (0)
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:28 Instance name is server2.
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: 2023/07/19 14:47:28 Current master is .
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:47:28  ag(ag_cluster)[55236]:    INFO: ag_cluster start : 0
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: mssql_validate
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: mssql_notify post-start
>Jul 19 14:47:28  ag(ag_cluster)[55435]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: mssql_validate
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: mssql_notify pre-promote
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: notify: 2023/07/19 14:47:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: notify: 2023/07/19 14:47:28 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:28  ag(ag_cluster)[55481]:    INFO: notify: 2023/07/19 14:47:28 Connected to the instance at localhost:1433
>Jul 19 14:47:33  ag(ag_cluster)[55481]:    INFO: notify: 2023/07/19 14:47:33 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:47:33  ag(ag_cluster)[55481]:    INFO: notify: 2023/07/19 14:47:33 Sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:33  ag(ag_cluster)[55481]:    INFO: notify: SEQUENCE_NUMBER: 61899068669959
>Jul 19 14:47:38  ag(ag_cluster)[55481]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: mssql_validate
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: mssql_notify pre-promote
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: notify: 2023/07/19 14:47:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: notify: 2023/07/19 14:47:38 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:38  ag(ag_cluster)[55587]:    INFO: notify: 2023/07/19 14:47:38 Connected to the instance at localhost:1433
>Jul 19 14:47:43  ag(ag_cluster)[55587]:    INFO: notify: 2023/07/19 14:47:43 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:47:43  ag(ag_cluster)[55587]:    INFO: notify: 2023/07/19 14:47:43 Sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:43  ag(ag_cluster)[55587]:    INFO: notify: SEQUENCE_NUMBER: 61899068669959
>Jul 19 14:47:48  ag(ag_cluster)[55587]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: mssql_validate
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: Resource agent invoked with: promote
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: mssql_promote
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:49 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:49 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:49  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:49 Connected to the instance at localhost:1433
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Replica is RESOLVING (0)
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Line does not match expected syntax. Ignoring.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61899068669959"]
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61899068669959"]
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 2 sequence numbers were found
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Max sequence number is 384C:7 (61899068669959)
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Sequence number of server2 is 384C:7 (61899068669959)
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Promoting replica to PRIMARY role...
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Replica is PRIMARY (1)
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Replica is now PRIMARY
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Monitor Caller is: promote.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Replica is PRIMARY (1)
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Instance name is server2.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Current master is server2.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 DB_FAILOVER is ON.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: 2023/07/19 14:47:54 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:47:54  ag(ag_cluster)[55707]:    INFO: ag_cluster promote : 0
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: mssql_validate
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: Resource agent invoked with: notify
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: mssql_notify post-promote
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: notify: 2023/07/19 14:47:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: notify: 2023/07/19 14:47:54 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: notify: 2023/07/19 14:47:54 Replica is PRIMARY (1)
>Jul 19 14:47:54  ag(ag_cluster)[55980]:    INFO: ag_cluster notify : 0
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: mssql_validate
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:47:54  IPaddr2(virtualip)[56063]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: mssql_monitor
>Jul 19 14:47:54  IPaddr2(virtualip)[56063]:    INFO: Bringing device eth0 up
>Jul 19 14:47:54  IPaddr2(virtualip)[56063]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:54 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:54 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:54 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:47:54  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:54 Connected to the instance at localhost:1433
>Jul 19 14:47:58  IPaddr2(virtualip)[56063]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 Monitor Caller is: monitor.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 Replica is PRIMARY (1)
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 Instance name is server2.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 Current master is server2.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 DB_FAILOVER is ON.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: 2023/07/19 14:47:59 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:47:59  ag(ag_cluster)[56056]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:48:00  ag(ag_cluster)[56056]:    INFO: ag_cluster monitor : 8
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: mssql_validate
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: mssql_notify pre-stop
>Jul 19 14:48:06  ag(ag_cluster)[56410]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: mssql_validate
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: mssql_notify pre-stop
>Jul 19 14:48:06  ag(ag_cluster)[56455]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: mssql_validate
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: mssql_notify post-stop
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: notify: 2023/07/19 14:48:06 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: notify: 2023/07/19 14:48:06 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: notify: 2023/07/19 14:48:06 Replica is PRIMARY (1)
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: notify: 2023/07/19 14:48:06 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: notify: 2023/07/19 14:48:06 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:48:06  ag(ag_cluster)[56500]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:06  ag(ag_cluster)[56587]:    INFO: mssql_validate
>Jul 19 14:48:06  ag(ag_cluster)[56587]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:06  ag(ag_cluster)[56587]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: mssql_notify pre-start
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 Replica is PRIMARY (1)
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: 2023/07/19 14:48:07 Sequence number is 384D:6 (61903363637254)
>Jul 19 14:48:07  ag(ag_cluster)[56587]:    INFO: notify: SEQUENCE_NUMBER: 61903363637254
>Jul 19 14:48:12  ag(ag_cluster)[56587]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: mssql_validate
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: mssql_monitor
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:12 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:12 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:12 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:48:12  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:12 Connected to the instance at localhost:1433
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 Monitor Caller is: monitor.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 Replica is PRIMARY (1)
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 Instance name is server2.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 Current master is server2.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 DB_FAILOVER is ON.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: 2023/07/19 14:48:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:48:17  ag(ag_cluster)[56700]:    INFO: ag_cluster monitor : 8
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: mssql_validate
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: mssql_notify pre-start
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 Replica is PRIMARY (1)
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: 2023/07/19 14:48:17 Sequence number is 384D:7 (61903363637255)
>Jul 19 14:48:17  ag(ag_cluster)[56927]:    INFO: notify: SEQUENCE_NUMBER: 61903363637255
>Jul 19 14:48:22  ag(ag_cluster)[56927]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:22  ag(ag_cluster)[57040]:    INFO: mssql_validate
>Jul 19 14:48:22  ag(ag_cluster)[57040]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:22  ag(ag_cluster)[57040]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: mssql_notify pre-start
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 Replica is PRIMARY (1)
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: 2023/07/19 14:48:23 Sequence number is 384D:7 (61903363637255)
>Jul 19 14:48:23  ag(ag_cluster)[57040]:    INFO: notify: SEQUENCE_NUMBER: 61903363637255
>Jul 19 14:48:28  ag(ag_cluster)[57040]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: mssql_validate
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: mssql_monitor
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:28 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:48:28  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:28 Connected to the instance at localhost:1433
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Monitor Caller is: monitor.
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Replica is PRIMARY (1)
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Offlining replica...
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Replica is RESOLVING (0)
>Jul 19 14:48:33  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Instance name is server2.
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Current master is server1
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: server2.
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Setting the role to Secondary.
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Setting replica to SECONDARY role...
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: 2023/07/19 14:48:33 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:48:34  ag(ag_cluster)[57195]:    INFO: ag_cluster monitor : 0
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: mssql_notify post-start
>Jul 19 14:48:34  ag(ag_cluster)[57391]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: mssql_notify pre-demote
>Jul 19 14:48:34  ag(ag_cluster)[57439]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:34  IPaddr2(virtualip)[57438]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: Resource agent invoked with: demote
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: mssql_demote
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: demote: 2023/07/19 14:48:34 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: demote: 2023/07/19 14:48:34 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: demote: 2023/07/19 14:48:34 Replica is RESOLVING (0)
>Jul 19 14:48:34  ag(ag_cluster)[57537]:    INFO: ag_cluster demote : 0
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: mssql_notify post-demote
>Jul 19 14:48:34  ag(ag_cluster)[57630]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: mssql_notify pre-stop
>Jul 19 14:48:34  ag(ag_cluster)[57675]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: mssql_validate
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: Resource agent invoked with: stop
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: mssql_stop
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: stop: 2023/07/19 14:48:34 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: stop: 2023/07/19 14:48:34 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: stop: 2023/07/19 14:48:34 Replica is RESOLVING (0)
>Jul 19 14:48:34  ag(ag_cluster)[57720]:    INFO: ag_cluster stop : 0
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: mssql_validate
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: Resource agent invoked with: start
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: mssql_start
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:45 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:45 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:48:45  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:45 Connected to the instance at localhost:1433
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Replica is RESOLVING (0)
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Sequence number is 384D:7 (61903363637255)
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Line does not match expected syntax. Ignoring.
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61903363637255"]
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Line does not match expected syntax. Ignoring.
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 1 sequence numbers were found
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Max sequence number is 384D:7 (61903363637255)
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Monitor Caller is: start.
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Replica is RESOLVING (0)
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Instance name is server2.
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: 2023/07/19 14:48:50 Current master is .
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:48:50  ag(ag_cluster)[57814]:    INFO: ag_cluster start : 0
>Jul 19 14:48:50  ag(ag_cluster)[58014]:    INFO: mssql_validate
>Jul 19 14:48:50  ag(ag_cluster)[58014]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:50  ag(ag_cluster)[58014]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:51  ag(ag_cluster)[58014]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:51  ag(ag_cluster)[58014]:    INFO: mssql_notify post-start
>Jul 19 14:48:51  ag(ag_cluster)[58014]:    INFO: ag_cluster notify : 0
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: mssql_validate
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: Resource agent invoked with: notify
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: mssql_notify pre-promote
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: notify: 2023/07/19 14:48:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: notify: 2023/07/19 14:48:51 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:48:51  ag(ag_cluster)[58060]:    INFO: notify: 2023/07/19 14:48:51 Connected to the instance at localhost:1433
>Jul 19 14:48:56  ag(ag_cluster)[58060]:    INFO: notify: 2023/07/19 14:48:56 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:48:56  ag(ag_cluster)[58060]:    INFO: notify: 2023/07/19 14:48:56 Sequence number is 384D:7 (61903363637255)
>Jul 19 14:48:56  ag(ag_cluster)[58060]:    INFO: notify: SEQUENCE_NUMBER: 61903363637255
>Jul 19 14:49:01  ag(ag_cluster)[58060]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: mssql_validate
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: mssql_notify pre-promote
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: notify: 2023/07/19 14:49:01 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: notify: 2023/07/19 14:49:01 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:49:01  ag(ag_cluster)[58167]:    INFO: notify: 2023/07/19 14:49:01 Connected to the instance at localhost:1433
>Jul 19 14:49:06  ag(ag_cluster)[58167]:    INFO: notify: 2023/07/19 14:49:06 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:49:06  ag(ag_cluster)[58167]:    INFO: notify: 2023/07/19 14:49:06 Sequence number is 384D:7 (61903363637255)
>Jul 19 14:49:06  ag(ag_cluster)[58167]:    INFO: notify: SEQUENCE_NUMBER: 61903363637255
>Jul 19 14:49:11  ag(ag_cluster)[58167]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: mssql_validate
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: Resource agent invoked with: promote
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: mssql_promote
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:11 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:11 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:11 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:49:11  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:11 Connected to the instance at localhost:1433
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Replica is RESOLVING (0)
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Line does not match expected syntax. Ignoring.
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61903363637255"]
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61903363637255"]
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 2 sequence numbers were found
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Max sequence number is 384D:7 (61903363637255)
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Sequence number of server2 is 384D:7 (61903363637255)
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Promoting replica to PRIMARY role...
>Jul 19 14:49:16  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Replica is PRIMARY (1)
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Replica is now PRIMARY
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Monitor Caller is: promote.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Replica is PRIMARY (1)
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Instance name is server2.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Current master is server2.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 DB_FAILOVER is ON.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: 2023/07/19 14:49:16 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:49:17  ag(ag_cluster)[58276]:    INFO: ag_cluster promote : 0
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: mssql_validate
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: mssql_notify post-promote
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: notify: 2023/07/19 14:49:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: notify: 2023/07/19 14:49:17 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: notify: 2023/07/19 14:49:17 Replica is PRIMARY (1)
>Jul 19 14:49:17  ag(ag_cluster)[58551]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: mssql_validate
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: mssql_monitor
>Jul 19 14:49:17  IPaddr2(virtualip)[58635]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:49:17  IPaddr2(virtualip)[58635]:    INFO: Bringing device eth0 up
>Jul 19 14:49:17  IPaddr2(virtualip)[58635]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:17 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:49:17  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:17 Connected to the instance at localhost:1433
>Jul 19 14:49:21  IPaddr2(virtualip)[58635]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 Monitor Caller is: monitor.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 Replica is PRIMARY (1)
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 Instance name is server2.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 Current master is server2.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 DB_FAILOVER is ON.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: 2023/07/19 14:49:22 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:49:22  ag(ag_cluster)[58627]:    INFO: ag_cluster monitor : 8
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: mssql_validate
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: mssql_notify pre-stop
>Jul 19 14:49:29  ag(ag_cluster)[58982]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: mssql_validate
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: mssql_notify pre-stop
>Jul 19 14:49:29  ag(ag_cluster)[59027]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: mssql_validate
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: mssql_notify post-stop
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: notify: 2023/07/19 14:49:29 Replica is PRIMARY (1)
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: notify: 2023/07/19 14:49:29 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: notify: 2023/07/19 14:49:29 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:29  ag(ag_cluster)[59072]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: mssql_validate
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: mssql_notify pre-start
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 Replica is PRIMARY (1)
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: 2023/07/19 14:49:29 Sequence number is 384E:6 (61907658604550)
>Jul 19 14:49:29  ag(ag_cluster)[59159]:    INFO: notify: SEQUENCE_NUMBER: 61907658604550
>Jul 19 14:49:34  ag(ag_cluster)[59159]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: mssql_validate
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: mssql_monitor
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:35 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:35 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:49:35  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:35 Connected to the instance at localhost:1433
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 Monitor Caller is: monitor.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 Replica is PRIMARY (1)
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 Instance name is server2.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 Current master is server2.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 DB_FAILOVER is ON.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: 2023/07/19 14:49:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:49:40  ag(ag_cluster)[59271]:    INFO: ag_cluster monitor : 8
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: mssql_validate
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: mssql_notify pre-start
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 Replica is PRIMARY (1)
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: 2023/07/19 14:49:40 Sequence number is 384E:B (61907658604555)
>Jul 19 14:49:40  ag(ag_cluster)[59498]:    INFO: notify: SEQUENCE_NUMBER: 61907658604555
>Jul 19 14:49:45  ag(ag_cluster)[59498]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: mssql_validate
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: mssql_notify pre-start
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 Replica is PRIMARY (1)
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: 2023/07/19 14:49:45 Sequence number is 384E:B (61907658604555)
>Jul 19 14:49:45  ag(ag_cluster)[59618]:    INFO: notify: SEQUENCE_NUMBER: 61907658604555
>Jul 19 14:49:50  ag(ag_cluster)[59618]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: mssql_validate
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: mssql_monitor
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:51 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:51 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:51 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:49:51  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:51 Connected to the instance at localhost:1433
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Monitor Caller is: monitor.
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Replica is PRIMARY (1)
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Offlining replica...
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Replica is RESOLVING (0)
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Instance name is server2.
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Current master is server1
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: server2.
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Setting the role to Secondary.
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Setting replica to SECONDARY role...
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: 2023/07/19 14:49:56 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:49:56  ag(ag_cluster)[59774]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:49:57  ag(ag_cluster)[59774]:    INFO: ag_cluster monitor : 0
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: mssql_notify post-start
>Jul 19 14:49:57  ag(ag_cluster)[59972]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:57  IPaddr2(virtualip)[60018]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: mssql_notify pre-demote
>Jul 19 14:49:57  ag(ag_cluster)[60019]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: Resource agent invoked with: demote
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: mssql_demote
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: demote: 2023/07/19 14:49:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: demote: 2023/07/19 14:49:57 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: demote: 2023/07/19 14:49:57 Replica is RESOLVING (0)
>Jul 19 14:49:57  ag(ag_cluster)[60117]:    INFO: ag_cluster demote : 0
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: mssql_notify post-demote
>Jul 19 14:49:57  ag(ag_cluster)[60211]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: Resource agent invoked with: notify
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: mssql_notify pre-stop
>Jul 19 14:49:57  ag(ag_cluster)[60256]:    INFO: ag_cluster notify : 0
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: mssql_validate
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: Resource agent invoked with: stop
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: mssql_stop
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: stop: 2023/07/19 14:49:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: stop: 2023/07/19 14:49:57 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: stop: 2023/07/19 14:49:57 Replica is RESOLVING (0)
>Jul 19 14:49:57  ag(ag_cluster)[60301]:    INFO: ag_cluster stop : 0
>Jul 19 14:50:08  ag(ag_cluster)[60388]:    INFO: mssql_validate
>Jul 19 14:50:08  ag(ag_cluster)[60388]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:08  ag(ag_cluster)[60388]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:08  ag(ag_cluster)[60388]:    INFO: Resource agent invoked with: start
>Jul 19 14:50:08  ag(ag_cluster)[60388]:    INFO: mssql_start
>Jul 19 14:50:09  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:08 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:50:09  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:08 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:50:09  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:08 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:09  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:09 Connected to the instance at localhost:1433
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Replica is RESOLVING (0)
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Sequence number is 384E:B (61907658604555)
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Line does not match expected syntax. Ignoring.
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61907658604555"]
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Line does not match expected syntax. Ignoring.
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 1 sequence numbers were found
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Max sequence number is 384E:B (61907658604555)
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Monitor Caller is: start.
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Replica is RESOLVING (0)
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Instance name is server2.
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: 2023/07/19 14:50:14 Current master is .
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:50:14  ag(ag_cluster)[60388]:    INFO: ag_cluster start : 0
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: mssql_validate
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: mssql_notify post-start
>Jul 19 14:50:14  ag(ag_cluster)[60587]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: mssql_validate
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: mssql_notify pre-promote
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: notify: 2023/07/19 14:50:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: notify: 2023/07/19 14:50:14 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:14  ag(ag_cluster)[60633]:    INFO: notify: 2023/07/19 14:50:14 Connected to the instance at localhost:1433
>Jul 19 14:50:19  ag(ag_cluster)[60633]:    INFO: notify: 2023/07/19 14:50:19 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:50:19  ag(ag_cluster)[60633]:    INFO: notify: 2023/07/19 14:50:19 Sequence number is 384E:B (61907658604555)
>Jul 19 14:50:19  ag(ag_cluster)[60633]:    INFO: notify: SEQUENCE_NUMBER: 61907658604555
>Jul 19 14:50:24  ag(ag_cluster)[60633]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: mssql_validate
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: mssql_notify pre-promote
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: notify: 2023/07/19 14:50:24 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: notify: 2023/07/19 14:50:24 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:24  ag(ag_cluster)[60739]:    INFO: notify: 2023/07/19 14:50:24 Connected to the instance at localhost:1433
>Jul 19 14:50:29  ag(ag_cluster)[60739]:    INFO: notify: 2023/07/19 14:50:29 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:50:30  ag(ag_cluster)[60739]:    INFO: notify: 2023/07/19 14:50:29 Sequence number is 384E:B (61907658604555)
>Jul 19 14:50:30  ag(ag_cluster)[60739]:    INFO: notify: SEQUENCE_NUMBER: 61907658604555
>Jul 19 14:50:35  ag(ag_cluster)[60739]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: mssql_validate
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: Resource agent invoked with: promote
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: mssql_promote
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:35 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:35 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:35 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:35  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:35 Connected to the instance at localhost:1433
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Replica is RESOLVING (0)
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Line does not match expected syntax. Ignoring.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61907658604555"]
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61907658604555"]
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 2 sequence numbers were found
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Max sequence number is 384E:B (61907658604555)
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Sequence number of server2 is 384E:B (61907658604555)
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Promoting replica to PRIMARY role...
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Replica is PRIMARY (1)
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Replica is now PRIMARY
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Monitor Caller is: promote.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Replica is PRIMARY (1)
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Instance name is server2.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Current master is server2.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 DB_FAILOVER is ON.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: 2023/07/19 14:50:40 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:50:40  ag(ag_cluster)[60848]:    INFO: ag_cluster promote : 0
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: mssql_validate
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: mssql_notify post-promote
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: notify: 2023/07/19 14:50:40 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: notify: 2023/07/19 14:50:40 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: notify: 2023/07/19 14:50:40 Replica is PRIMARY (1)
>Jul 19 14:50:40  ag(ag_cluster)[61120]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: mssql_validate
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: mssql_monitor
>Jul 19 14:50:41  IPaddr2(virtualip)[61206]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:50:41  IPaddr2(virtualip)[61206]:    INFO: Bringing device eth0 up
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:50:41  IPaddr2(virtualip)[61206]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:41 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:41  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:41 Connected to the instance at localhost:1433
>Jul 19 14:50:45  IPaddr2(virtualip)[61206]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 Monitor Caller is: monitor.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 Replica is PRIMARY (1)
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 Instance name is server2.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 Current master is server2.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 DB_FAILOVER is ON.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: 2023/07/19 14:50:46 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:50:46  ag(ag_cluster)[61196]:    INFO: ag_cluster monitor : 8
>Jul 19 14:50:52  ag(ag_cluster)[61555]:    INFO: mssql_validate
>Jul 19 14:50:52  ag(ag_cluster)[61555]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:52  ag(ag_cluster)[61555]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:52  ag(ag_cluster)[61555]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:52  ag(ag_cluster)[61555]:    INFO: mssql_notify pre-stop
>Jul 19 14:50:53  ag(ag_cluster)[61555]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: mssql_validate
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: mssql_notify pre-stop
>Jul 19 14:50:53  ag(ag_cluster)[61600]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: mssql_validate
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: mssql_notify post-stop
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: notify: 2023/07/19 14:50:53 Replica is PRIMARY (1)
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: notify: 2023/07/19 14:50:53 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: notify: 2023/07/19 14:50:53 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:50:53  ag(ag_cluster)[61645]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: mssql_validate
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: Resource agent invoked with: notify
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: mssql_notify pre-start
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 Replica is PRIMARY (1)
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: 2023/07/19 14:50:53 Sequence number is 384F:6 (61911953571846)
>Jul 19 14:50:53  ag(ag_cluster)[61732]:    INFO: notify: SEQUENCE_NUMBER: 61911953571846
>Jul 19 14:50:58  ag(ag_cluster)[61732]:    INFO: ag_cluster notify : 0
>Jul 19 14:50:58  ag(ag_cluster)[61843]:    INFO: mssql_validate
>Jul 19 14:50:58  ag(ag_cluster)[61843]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:50:58  ag(ag_cluster)[61843]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:50:58  ag(ag_cluster)[61843]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:50:59  ag(ag_cluster)[61843]:    INFO: mssql_monitor
>Jul 19 14:50:59  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:50:59 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:50:59  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:50:59 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:50:59  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:50:59 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:50:59  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:50:59 Connected to the instance at localhost:1433
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 Monitor Caller is: monitor.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 Replica is PRIMARY (1)
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 Instance name is server2.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 Current master is server2.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 DB_FAILOVER is ON.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: 2023/07/19 14:51:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:51:04  ag(ag_cluster)[61843]:    INFO: ag_cluster monitor : 8
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: mssql_validate
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: mssql_notify pre-start
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 Replica is PRIMARY (1)
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: 2023/07/19 14:51:04 Sequence number is 384F:B (61911953571851)
>Jul 19 14:51:04  ag(ag_cluster)[62069]:    INFO: notify: SEQUENCE_NUMBER: 61911953571851
>Jul 19 14:51:09  ag(ag_cluster)[62069]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: mssql_validate
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: mssql_notify pre-start
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:51:09  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:09 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:10 Replica is PRIMARY (1)
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:10 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:10 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:10 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: 2023/07/19 14:51:10 Sequence number is 384F:B (61911953571851)
>Jul 19 14:51:10  ag(ag_cluster)[62182]:    INFO: notify: SEQUENCE_NUMBER: 61911953571851
>Jul 19 14:51:15  ag(ag_cluster)[62182]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: mssql_validate
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: mssql_monitor
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:15 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:51:15  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:15 Connected to the instance at localhost:1433
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Monitor Caller is: monitor.
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Replica is PRIMARY (1)
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Offlining replica...
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Replica is RESOLVING (0)
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Instance name is server2.
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Current master is server1
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: server2.
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Setting the role to Secondary.
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Setting replica to SECONDARY role...
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: 2023/07/19 14:51:20 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:51:20  ag(ag_cluster)[62335]:    INFO: ag_cluster monitor : 0
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: mssql_validate
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: mssql_notify post-start
>Jul 19 14:51:20  ag(ag_cluster)[62531]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: mssql_validate
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:21  IPaddr2(virtualip)[62577]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: mssql_notify pre-demote
>Jul 19 14:51:21  ag(ag_cluster)[62578]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: mssql_validate
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: Resource agent invoked with: demote
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: mssql_demote
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: demote: 2023/07/19 14:51:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: demote: 2023/07/19 14:51:21 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: demote: 2023/07/19 14:51:21 Replica is RESOLVING (0)
>Jul 19 14:51:21  ag(ag_cluster)[62676]:    INFO: ag_cluster demote : 0
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: mssql_validate
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: mssql_notify post-demote
>Jul 19 14:51:21  ag(ag_cluster)[62769]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: mssql_validate
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: mssql_notify pre-stop
>Jul 19 14:51:21  ag(ag_cluster)[62814]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: mssql_validate
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: Resource agent invoked with: stop
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: mssql_stop
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: stop: 2023/07/19 14:51:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: stop: 2023/07/19 14:51:21 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: stop: 2023/07/19 14:51:21 Replica is RESOLVING (0)
>Jul 19 14:51:21  ag(ag_cluster)[62859]:    INFO: ag_cluster stop : 0
>Jul 19 14:51:31  ag(ag_cluster)[62941]:    INFO: mssql_validate
>Jul 19 14:51:31  ag(ag_cluster)[62941]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:31  ag(ag_cluster)[62941]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: Resource agent invoked with: start
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: mssql_start
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:32 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:32 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:51:32  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:32 Connected to the instance at localhost:1433
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Replica is RESOLVING (0)
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Sequence number is 384F:B (61911953571851)
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Line does not match expected syntax. Ignoring.
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61911953571851"]
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Line does not match expected syntax. Ignoring.
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 1 sequence numbers were found
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Max sequence number is 384F:B (61911953571851)
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Monitor Caller is: start.
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Replica is RESOLVING (0)
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Instance name is server2.
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: 2023/07/19 14:51:37 Current master is .
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:51:37  ag(ag_cluster)[62941]:    INFO: ag_cluster start : 0
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: mssql_validate
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: mssql_notify post-start
>Jul 19 14:51:37  ag(ag_cluster)[63138]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: mssql_validate
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: mssql_notify pre-promote
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: notify: 2023/07/19 14:51:37 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: notify: 2023/07/19 14:51:37 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:51:37  ag(ag_cluster)[63184]:    INFO: notify: 2023/07/19 14:51:37 Connected to the instance at localhost:1433
>Jul 19 14:51:42  ag(ag_cluster)[63184]:    INFO: notify: 2023/07/19 14:51:42 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:51:42  ag(ag_cluster)[63184]:    INFO: notify: 2023/07/19 14:51:42 Sequence number is 384F:B (61911953571851)
>Jul 19 14:51:42  ag(ag_cluster)[63184]:    INFO: notify: SEQUENCE_NUMBER: 61911953571851
>Jul 19 14:51:47  ag(ag_cluster)[63184]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: mssql_validate
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: Resource agent invoked with: notify
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: mssql_notify pre-promote
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: notify: 2023/07/19 14:51:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: notify: 2023/07/19 14:51:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:51:48  ag(ag_cluster)[63296]:    INFO: notify: 2023/07/19 14:51:48 Connected to the instance at localhost:1433
>Jul 19 14:51:53  ag(ag_cluster)[63296]:    INFO: notify: 2023/07/19 14:51:53 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:51:53  ag(ag_cluster)[63296]:    INFO: notify: 2023/07/19 14:51:53 Sequence number is 384F:B (61911953571851)
>Jul 19 14:51:53  ag(ag_cluster)[63296]:    INFO: notify: SEQUENCE_NUMBER: 61911953571851
>Jul 19 14:51:58  ag(ag_cluster)[63296]:    INFO: ag_cluster notify : 0
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: mssql_validate
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: Resource agent invoked with: promote
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: mssql_promote
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:51:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:51:58 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:51:58 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:51:58  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:51:58 Connected to the instance at localhost:1433
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Replica is RESOLVING (0)
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Line does not match expected syntax. Ignoring.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61911953571851"]
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61911953571851"]
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 2 sequence numbers were found
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Max sequence number is 384F:B (61911953571851)
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Sequence number of server2 is 384F:B (61911953571851)
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Promoting replica to PRIMARY role...
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Replica is PRIMARY (1)
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Replica is now PRIMARY
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Monitor Caller is: promote.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Replica is PRIMARY (1)
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Instance name is server2.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Current master is server2.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 DB_FAILOVER is ON.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: 2023/07/19 14:52:03 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:52:03  ag(ag_cluster)[63404]:    INFO: ag_cluster promote : 0
>Jul 19 14:52:03  ag(ag_cluster)[63678]:    INFO: mssql_validate
>Jul 19 14:52:03  ag(ag_cluster)[63678]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:03  ag(ag_cluster)[63678]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:03  ag(ag_cluster)[63678]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:04  ag(ag_cluster)[63678]:    INFO: mssql_notify post-promote
>Jul 19 14:52:04  ag(ag_cluster)[63678]:    INFO: notify: 2023/07/19 14:52:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:52:04  ag(ag_cluster)[63678]:    INFO: notify: 2023/07/19 14:52:04 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:04  ag(ag_cluster)[63678]:    INFO: notify: 2023/07/19 14:52:04 Replica is PRIMARY (1)
>Jul 19 14:52:04  ag(ag_cluster)[63678]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: mssql_validate
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: mssql_monitor
>Jul 19 14:52:04  IPaddr2(virtualip)[63763]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:52:04  IPaddr2(virtualip)[63763]:    INFO: Bringing device eth0 up
>Jul 19 14:52:04  IPaddr2(virtualip)[63763]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:04 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:52:04  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:04 Connected to the instance at localhost:1433
>Jul 19 14:52:08  IPaddr2(virtualip)[63763]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 Monitor Caller is: monitor.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 Replica is PRIMARY (1)
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 Instance name is server2.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 Current master is server2.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 DB_FAILOVER is ON.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: 2023/07/19 14:52:09 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:52:09  ag(ag_cluster)[63753]:    INFO: ag_cluster monitor : 8
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: mssql_validate
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: mssql_notify pre-stop
>Jul 19 14:52:17  ag(ag_cluster)[64108]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: mssql_validate
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: mssql_notify pre-stop
>Jul 19 14:52:17  ag(ag_cluster)[64153]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: mssql_validate
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: mssql_notify post-stop
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: notify: 2023/07/19 14:52:17 Replica is PRIMARY (1)
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: notify: 2023/07/19 14:52:17 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: notify: 2023/07/19 14:52:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:17  ag(ag_cluster)[64198]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: mssql_validate
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: mssql_notify pre-start
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 Replica is PRIMARY (1)
>Jul 19 14:52:17  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:18  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:18  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:52:18  ag(ag_cluster)[64285]:    INFO: notify: 2023/07/19 14:52:17 Sequence number is 3850:8 (61916248539144)
>Jul 19 14:52:18  ag(ag_cluster)[64285]:    INFO: notify: SEQUENCE_NUMBER: 61916248539144
>Jul 19 14:52:23  ag(ag_cluster)[64285]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: mssql_validate
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: mssql_monitor
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:23 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:23 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:52:23  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:23 Connected to the instance at localhost:1433
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 Monitor Caller is: monitor.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 Replica is PRIMARY (1)
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 Instance name is server2.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 Current master is server2.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 DB_FAILOVER is ON.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: 2023/07/19 14:52:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:52:28  ag(ag_cluster)[64398]:    INFO: ag_cluster monitor : 8
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: mssql_validate
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: mssql_notify pre-start
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 Replica is PRIMARY (1)
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: 2023/07/19 14:52:28 Sequence number is 3850:B (61916248539147)
>Jul 19 14:52:28  ag(ag_cluster)[64624]:    INFO: notify: SEQUENCE_NUMBER: 61916248539147
>Jul 19 14:52:33  ag(ag_cluster)[64624]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: mssql_validate
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: mssql_notify pre-start
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 Replica is PRIMARY (1)
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: 2023/07/19 14:52:33 Sequence number is 3850:B (61916248539147)
>Jul 19 14:52:33  ag(ag_cluster)[64735]:    INFO: notify: SEQUENCE_NUMBER: 61916248539147
>Jul 19 14:52:38  ag(ag_cluster)[64735]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: mssql_validate
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: mssql_monitor
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:39 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:39 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:39 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:52:39  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:39 Connected to the instance at localhost:1433
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Monitor Caller is: monitor.
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Replica is PRIMARY (1)
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Offlining replica...
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Replica is RESOLVING (0)
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Instance name is server2.
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Current master is server1
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: server2.
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Setting the role to Secondary.
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Setting replica to SECONDARY role...
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: 2023/07/19 14:52:44 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:52:44  ag(ag_cluster)[64889]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:52:45  ag(ag_cluster)[64889]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:52:45  ag(ag_cluster)[64889]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:52:45  ag(ag_cluster)[64889]:    INFO: ag_cluster monitor : 0
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: mssql_notify post-start
>Jul 19 14:52:45  ag(ag_cluster)[65127]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: mssql_notify pre-demote
>Jul 19 14:52:45  IPaddr2(virtualip)[65181]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:52:45  ag(ag_cluster)[65182]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: Resource agent invoked with: demote
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: mssql_demote
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: demote: 2023/07/19 14:52:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: demote: 2023/07/19 14:52:45 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: demote: 2023/07/19 14:52:45 Replica is RESOLVING (0)
>Jul 19 14:52:45  ag(ag_cluster)[65280]:    INFO: ag_cluster demote : 0
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: mssql_notify post-demote
>Jul 19 14:52:45  ag(ag_cluster)[65373]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: Resource agent invoked with: notify
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: mssql_notify pre-stop
>Jul 19 14:52:45  ag(ag_cluster)[65418]:    INFO: ag_cluster notify : 0
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: mssql_validate
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: Resource agent invoked with: stop
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: mssql_stop
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: stop: 2023/07/19 14:52:45 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:52:45  ag(ag_cluster)[65463]:    INFO: stop: 2023/07/19 14:52:45 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:52:46  ag(ag_cluster)[65463]:    INFO: stop: 2023/07/19 14:52:45 Replica is RESOLVING (0)
>Jul 19 14:52:46  ag(ag_cluster)[65463]:    INFO: ag_cluster stop : 0
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: mssql_validate
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: Resource agent invoked with: start
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: mssql_start
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:52:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:52:57 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:52:57 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:52:57  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:52:57 Connected to the instance at localhost:1433
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Replica is RESOLVING (0)
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Sequence number is 3850:B (61916248539147)
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Line does not match expected syntax. Ignoring.
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61916248539147"]
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Line does not match expected syntax. Ignoring.
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 1 sequence numbers were found
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Max sequence number is 3850:B (61916248539147)
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Monitor Caller is: start.
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Replica is RESOLVING (0)
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Instance name is server2.
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: 2023/07/19 14:53:02 Current master is .
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:53:02  ag(ag_cluster)[65545]:    INFO: ag_cluster start : 0
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: mssql_validate
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: mssql_notify post-start
>Jul 19 14:53:02  ag(ag_cluster)[65744]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: mssql_validate
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: mssql_notify pre-promote
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: notify: 2023/07/19 14:53:02 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: notify: 2023/07/19 14:53:02 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:53:02  ag(ag_cluster)[65790]:    INFO: notify: 2023/07/19 14:53:02 Connected to the instance at localhost:1433
>Jul 19 14:53:07  ag(ag_cluster)[65790]:    INFO: notify: 2023/07/19 14:53:07 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:53:07  ag(ag_cluster)[65790]:    INFO: notify: 2023/07/19 14:53:07 Sequence number is 3850:B (61916248539147)
>Jul 19 14:53:07  ag(ag_cluster)[65790]:    INFO: notify: SEQUENCE_NUMBER: 61916248539147
>Jul 19 14:53:13  ag(ag_cluster)[65790]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: mssql_validate
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: mssql_notify pre-promote
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: notify: 2023/07/19 14:53:13 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: notify: 2023/07/19 14:53:13 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:53:13  ag(ag_cluster)[65894]:    INFO: notify: 2023/07/19 14:53:13 Connected to the instance at localhost:1433
>Jul 19 14:53:18  ag(ag_cluster)[65894]:    INFO: notify: 2023/07/19 14:53:18 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:53:18  ag(ag_cluster)[65894]:    INFO: notify: 2023/07/19 14:53:18 Sequence number is 3850:B (61916248539147)
>Jul 19 14:53:18  ag(ag_cluster)[65894]:    INFO: notify: SEQUENCE_NUMBER: 61916248539147
>Jul 19 14:53:23  ag(ag_cluster)[65894]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: mssql_validate
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: Resource agent invoked with: promote
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: mssql_promote
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:23 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:23 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:23 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:53:23  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:23 Connected to the instance at localhost:1433
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Replica is RESOLVING (0)
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Line does not match expected syntax. Ignoring.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61916248539147"]
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61916248539147"]
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 2 sequence numbers were found
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Max sequence number is 3850:B (61916248539147)
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Sequence number of server2 is 3850:B (61916248539147)
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Promoting replica to PRIMARY role...
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Replica is PRIMARY (1)
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Replica is now PRIMARY
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Monitor Caller is: promote.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Replica is PRIMARY (1)
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Instance name is server2.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Current master is server2.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 DB_FAILOVER is ON.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: 2023/07/19 14:53:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:53:28  ag(ag_cluster)[65998]:    INFO: ag_cluster promote : 0
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: mssql_validate
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: mssql_notify post-promote
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: notify: 2023/07/19 14:53:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: notify: 2023/07/19 14:53:29 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: notify: 2023/07/19 14:53:29 Replica is PRIMARY (1)
>Jul 19 14:53:29  ag(ag_cluster)[66270]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: mssql_validate
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: mssql_monitor
>Jul 19 14:53:29  IPaddr2(virtualip)[66360]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:53:29  IPaddr2(virtualip)[66360]:    INFO: Bringing device eth0 up
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:29 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:53:29  IPaddr2(virtualip)[66360]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:29 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:29 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:53:29  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:29 Connected to the instance at localhost:1433
>Jul 19 14:53:33  IPaddr2(virtualip)[66360]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 Monitor Caller is: monitor.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 Replica is PRIMARY (1)
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 Instance name is server2.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 Current master is server2.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 DB_FAILOVER is ON.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: 2023/07/19 14:53:34 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:53:34  ag(ag_cluster)[66345]:    INFO: ag_cluster monitor : 8
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: mssql_validate
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: mssql_notify pre-stop
>Jul 19 14:53:41  ag(ag_cluster)[66700]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: mssql_validate
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: mssql_notify pre-stop
>Jul 19 14:53:41  ag(ag_cluster)[66746]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: mssql_validate
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: mssql_notify post-stop
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: notify: 2023/07/19 14:53:41 Replica is PRIMARY (1)
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: notify: 2023/07/19 14:53:41 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: notify: 2023/07/19 14:53:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:41  ag(ag_cluster)[66791]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: mssql_validate
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: mssql_notify pre-start
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 Replica is PRIMARY (1)
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: 2023/07/19 14:53:41 Sequence number is 3851:6 (61920543506438)
>Jul 19 14:53:41  ag(ag_cluster)[66877]:    INFO: notify: SEQUENCE_NUMBER: 61920543506438
>Jul 19 14:53:47  ag(ag_cluster)[66877]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: mssql_validate
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: mssql_monitor
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:47 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:47 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:53:47  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:47 Connected to the instance at localhost:1433
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 Monitor Caller is: monitor.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 Replica is PRIMARY (1)
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 Instance name is server2.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 Current master is server2.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 DB_FAILOVER is ON.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: 2023/07/19 14:53:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:53:52  ag(ag_cluster)[66996]:    INFO: ag_cluster monitor : 8
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: mssql_validate
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: mssql_notify pre-start
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 Replica is PRIMARY (1)
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: 2023/07/19 14:53:52 Sequence number is 3851:7 (61920543506439)
>Jul 19 14:53:52  ag(ag_cluster)[67223]:    INFO: notify: SEQUENCE_NUMBER: 61920543506439
>Jul 19 14:53:57  ag(ag_cluster)[67223]:    INFO: ag_cluster notify : 0
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: mssql_validate
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: Resource agent invoked with: notify
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: mssql_notify pre-start
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 Replica is PRIMARY (1)
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: 2023/07/19 14:53:57 Sequence number is 3851:7 (61920543506439)
>Jul 19 14:53:57  ag(ag_cluster)[67337]:    INFO: notify: SEQUENCE_NUMBER: 61920543506439
>Jul 19 14:54:02  ag(ag_cluster)[67337]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: mssql_validate
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: mssql_monitor
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:03 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:03 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:03 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:03  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:03 Connected to the instance at localhost:1433
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Monitor Caller is: monitor.
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Replica is PRIMARY (1)
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Offlining replica...
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Replica is RESOLVING (0)
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Instance name is server2.
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Current master is server1
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: server2.
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Setting the role to Secondary.
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Setting replica to SECONDARY role...
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: 2023/07/19 14:54:08 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:54:08  ag(ag_cluster)[67492]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:54:09  ag(ag_cluster)[67492]:    INFO: ag_cluster monitor : 0
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: mssql_notify post-start
>Jul 19 14:54:09  ag(ag_cluster)[67690]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:09  IPaddr2(virtualip)[67736]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: mssql_notify pre-demote
>Jul 19 14:54:09  ag(ag_cluster)[67737]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: Resource agent invoked with: demote
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: mssql_demote
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: demote: 2023/07/19 14:54:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: demote: 2023/07/19 14:54:09 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: demote: 2023/07/19 14:54:09 Replica is RESOLVING (0)
>Jul 19 14:54:09  ag(ag_cluster)[67835]:    INFO: ag_cluster demote : 0
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: mssql_notify post-demote
>Jul 19 14:54:09  ag(ag_cluster)[67928]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: mssql_notify pre-stop
>Jul 19 14:54:09  ag(ag_cluster)[67973]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: mssql_validate
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: Resource agent invoked with: stop
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: mssql_stop
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: stop: 2023/07/19 14:54:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: stop: 2023/07/19 14:54:09 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: stop: 2023/07/19 14:54:09 Replica is RESOLVING (0)
>Jul 19 14:54:09  ag(ag_cluster)[68018]:    INFO: ag_cluster stop : 0
>Jul 19 14:54:20  ag(ag_cluster)[68100]:    INFO: mssql_validate
>Jul 19 14:54:20  ag(ag_cluster)[68100]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:20  ag(ag_cluster)[68100]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:20  ag(ag_cluster)[68100]:    INFO: Resource agent invoked with: start
>Jul 19 14:54:21  ag(ag_cluster)[68100]:    INFO: mssql_start
>Jul 19 14:54:21  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:54:21  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:21 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:54:21  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:21 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:21  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:21 Connected to the instance at localhost:1433
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Replica is RESOLVING (0)
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Sequence number is 3851:7 (61920543506439)
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Line does not match expected syntax. Ignoring.
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61920543506439"]
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Line does not match expected syntax. Ignoring.
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 1 sequence numbers were found
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Max sequence number is 3851:7 (61920543506439)
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Monitor Caller is: start.
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Replica is RESOLVING (0)
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Instance name is server2.
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: 2023/07/19 14:54:26 Current master is .
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:54:26  ag(ag_cluster)[68100]:    INFO: ag_cluster start : 0
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: mssql_validate
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: mssql_notify post-start
>Jul 19 14:54:26  ag(ag_cluster)[68299]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: mssql_validate
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: mssql_notify pre-promote
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: notify: 2023/07/19 14:54:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: notify: 2023/07/19 14:54:26 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:26  ag(ag_cluster)[68345]:    INFO: notify: 2023/07/19 14:54:26 Connected to the instance at localhost:1433
>Jul 19 14:54:31  ag(ag_cluster)[68345]:    INFO: notify: 2023/07/19 14:54:31 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:54:31  ag(ag_cluster)[68345]:    INFO: notify: 2023/07/19 14:54:31 Sequence number is 3851:7 (61920543506439)
>Jul 19 14:54:31  ag(ag_cluster)[68345]:    INFO: notify: SEQUENCE_NUMBER: 61920543506439
>Jul 19 14:54:36  ag(ag_cluster)[68345]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: mssql_validate
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: mssql_notify pre-promote
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: notify: 2023/07/19 14:54:36 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: notify: 2023/07/19 14:54:36 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:36  ag(ag_cluster)[68448]:    INFO: notify: 2023/07/19 14:54:36 Connected to the instance at localhost:1433
>Jul 19 14:54:41  ag(ag_cluster)[68448]:    INFO: notify: 2023/07/19 14:54:41 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:54:41  ag(ag_cluster)[68448]:    INFO: notify: 2023/07/19 14:54:41 Sequence number is 3851:7 (61920543506439)
>Jul 19 14:54:41  ag(ag_cluster)[68448]:    INFO: notify: SEQUENCE_NUMBER: 61920543506439
>Jul 19 14:54:46  ag(ag_cluster)[68448]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: mssql_validate
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: Resource agent invoked with: promote
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: mssql_promote
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:47 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:47 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:47 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:47  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:47 Connected to the instance at localhost:1433
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Replica is RESOLVING (0)
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Line does not match expected syntax. Ignoring.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61920543506439"]
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61920543506439"]
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 2 sequence numbers were found
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Max sequence number is 3851:7 (61920543506439)
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Sequence number of server2 is 3851:7 (61920543506439)
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Promoting replica to PRIMARY role...
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Replica is PRIMARY (1)
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Replica is now PRIMARY
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Monitor Caller is: promote.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Replica is PRIMARY (1)
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Instance name is server2.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Current master is server2.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 DB_FAILOVER is ON.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: 2023/07/19 14:54:52 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:54:52  ag(ag_cluster)[68559]:    INFO: ag_cluster promote : 0
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: mssql_validate
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: Resource agent invoked with: notify
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: mssql_notify post-promote
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: notify: 2023/07/19 14:54:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: notify: 2023/07/19 14:54:52 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: notify: 2023/07/19 14:54:52 Replica is PRIMARY (1)
>Jul 19 14:54:52  ag(ag_cluster)[68830]:    INFO: ag_cluster notify : 0
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: mssql_validate
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: mssql_monitor
>Jul 19 14:54:52  IPaddr2(virtualip)[68916]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:54:52  IPaddr2(virtualip)[68916]:    INFO: Bringing device eth0 up
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:52 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:54:52  IPaddr2(virtualip)[68916]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:52 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:52 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:54:52  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:52 Connected to the instance at localhost:1433
>Jul 19 14:54:56  IPaddr2(virtualip)[68916]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 Monitor Caller is: monitor.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 Replica is PRIMARY (1)
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 Instance name is server2.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 Current master is server2.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 DB_FAILOVER is ON.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: 2023/07/19 14:54:57 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:54:57  ag(ag_cluster)[68905]:    INFO: ag_cluster monitor : 8
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: mssql_validate
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: mssql_notify pre-stop
>Jul 19 14:55:04  ag(ag_cluster)[69256]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: mssql_validate
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: mssql_notify pre-stop
>Jul 19 14:55:04  ag(ag_cluster)[69301]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: mssql_validate
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: mssql_notify post-stop
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: notify: 2023/07/19 14:55:04 Replica is PRIMARY (1)
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: notify: 2023/07/19 14:55:04 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: notify: 2023/07/19 14:55:04 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:55:04  ag(ag_cluster)[69346]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: mssql_validate
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: mssql_notify pre-start
>Jul 19 14:55:04  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:04 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:05 Replica is PRIMARY (1)
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:05 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:05 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:05 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: 2023/07/19 14:55:05 Sequence number is 3852:6 (61924838473734)
>Jul 19 14:55:05  ag(ag_cluster)[69433]:    INFO: notify: SEQUENCE_NUMBER: 61924838473734
>Jul 19 14:55:10  ag(ag_cluster)[69433]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: mssql_validate
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: mssql_monitor
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:10 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:10 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:10 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:55:10  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:10 Connected to the instance at localhost:1433
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 Monitor Caller is: monitor.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 Replica is PRIMARY (1)
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 Instance name is server2.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 Current master is server2.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 DB_FAILOVER is ON.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: 2023/07/19 14:55:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:55:15  ag(ag_cluster)[69545]:    INFO: ag_cluster monitor : 8
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: mssql_validate
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: mssql_notify pre-start
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 Replica is PRIMARY (1)
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: 2023/07/19 14:55:15 Sequence number is 3852:B (61924838473739)
>Jul 19 14:55:15  ag(ag_cluster)[69773]:    INFO: notify: SEQUENCE_NUMBER: 61924838473739
>Jul 19 14:55:20  ag(ag_cluster)[69773]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: mssql_validate
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: mssql_notify pre-start
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 Replica is PRIMARY (1)
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: 2023/07/19 14:55:21 Sequence number is 3852:B (61924838473739)
>Jul 19 14:55:21  ag(ag_cluster)[69884]:    INFO: notify: SEQUENCE_NUMBER: 61924838473739
>Jul 19 14:55:26  ag(ag_cluster)[69884]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: mssql_validate
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: mssql_monitor
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:26 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:26 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:26 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:55:26  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:26 Connected to the instance at localhost:1433
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Monitor Caller is: monitor.
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Replica is PRIMARY (1)
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Offlining replica...
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Replica is RESOLVING (0)
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Instance name is server2.
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Current master is server1
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: server2.
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Setting the role to Secondary.
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Setting replica to SECONDARY role...
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: 2023/07/19 14:55:31 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:55:31  ag(ag_cluster)[70038]:    INFO: ag_cluster monitor : 0
>Jul 19 14:55:31  ag(ag_cluster)[70233]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70233]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70233]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:32  ag(ag_cluster)[70233]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:32  ag(ag_cluster)[70233]:    INFO: mssql_notify post-start
>Jul 19 14:55:32  ag(ag_cluster)[70233]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:32  IPaddr2(virtualip)[70279]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: mssql_notify pre-demote
>Jul 19 14:55:32  ag(ag_cluster)[70280]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: Resource agent invoked with: demote
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: mssql_demote
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: demote: 2023/07/19 14:55:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: demote: 2023/07/19 14:55:32 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: demote: 2023/07/19 14:55:32 Replica is RESOLVING (0)
>Jul 19 14:55:32  ag(ag_cluster)[70378]:    INFO: ag_cluster demote : 0
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: mssql_notify post-demote
>Jul 19 14:55:32  ag(ag_cluster)[70471]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: mssql_notify pre-stop
>Jul 19 14:55:32  ag(ag_cluster)[70516]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: mssql_validate
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: Resource agent invoked with: stop
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: mssql_stop
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: stop: 2023/07/19 14:55:32 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: stop: 2023/07/19 14:55:32 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: stop: 2023/07/19 14:55:32 Replica is RESOLVING (0)
>Jul 19 14:55:32  ag(ag_cluster)[70561]:    INFO: ag_cluster stop : 0
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: mssql_validate
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: Resource agent invoked with: start
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: mssql_start
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-start]; connection-timeout [30]; health-threshold [3]; action [start]
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:43 ag-helper invoked with sequence-numbers [...]; required-synchronized-secondaries-to-commit [-1]; current-master []; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:43 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:55:43  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:43 Connected to the instance at localhost:1433
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 [DEBUG] AG Helper Start Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Replica is RESOLVING (0)
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Sequence number is 3852:B (61924838473739)
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Line does not match expected syntax. Ignoring.
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61924838473739"]
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Sequence number line [name="ag_cluster-sequence-number" host="server2" value=""]
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Line does not match expected syntax. Ignoring.
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 1 sequence numbers were found
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Max sequence number is 3852:B (61924838473739)
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Monitor Caller is: start.
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Replica is RESOLVING (0)
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Instance name is server2.
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: 2023/07/19 14:55:48 Current master is .
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: start: PROMOTION_SCORE: 10
>Jul 19 14:55:48  ag(ag_cluster)[70643]:    INFO: ag_cluster start : 0
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: mssql_validate
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: mssql_notify post-start
>Jul 19 14:55:48  ag(ag_cluster)[70849]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: mssql_validate
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: mssql_notify pre-promote
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: notify: 2023/07/19 14:55:48 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: notify: 2023/07/19 14:55:48 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:55:48  ag(ag_cluster)[70895]:    INFO: notify: 2023/07/19 14:55:48 Connected to the instance at localhost:1433
>Jul 19 14:55:53  ag(ag_cluster)[70895]:    INFO: notify: 2023/07/19 14:55:53 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:55:53  ag(ag_cluster)[70895]:    INFO: notify: 2023/07/19 14:55:53 Sequence number is 3852:B (61924838473739)
>Jul 19 14:55:53  ag(ag_cluster)[70895]:    INFO: notify: SEQUENCE_NUMBER: 61924838473739
>Jul 19 14:55:58  ag(ag_cluster)[70895]:    INFO: ag_cluster notify : 0
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: mssql_validate
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: Resource agent invoked with: notify
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: mssql_notify pre-promote
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: notify: 2023/07/19 14:55:58 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-promote]; connection-timeout [30]; health-threshold [3]; action [pre-promote]
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: notify: 2023/07/19 14:55:58 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:55:58  ag(ag_cluster)[70999]:    INFO: notify: 2023/07/19 14:55:58 Connected to the instance at localhost:1433
>Jul 19 14:56:03  ag(ag_cluster)[70999]:    INFO: notify: 2023/07/19 14:56:03 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:56:04  ag(ag_cluster)[70999]:    INFO: notify: 2023/07/19 14:56:03 Sequence number is 3852:B (61924838473739)
>Jul 19 14:56:04  ag(ag_cluster)[70999]:    INFO: notify: SEQUENCE_NUMBER: 61924838473739
>Jul 19 14:56:09  ag(ag_cluster)[70999]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: mssql_validate
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: Resource agent invoked with: promote
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: mssql_promote
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:09 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-promote]; connection-timeout [30]; health-threshold [3]; action [promote]
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:09 ag-helper invoked with skip-precheck [false]; sequence-numbers [...]; new-master [server2]; required-synchronized-secondaries-to-commit [-1]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [62]
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:09 From RetryExecute - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:56:09  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:09 Connected to the instance at localhost:1433
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Replica is RESOLVING (0)
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Verifying replica's sequence number vs all sequence numbers...
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server1" value=""]
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Line does not match expected syntax. Ignoring.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server3" value="61924838473739"]
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Sequence number line [name="ag_cluster-sequence-number" host="server2" value="61924838473739"]
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 2 sequence numbers were found
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Max sequence number is 3852:B (61924838473739)
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Sequence number of server2 is 3852:B (61924838473739)
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT or CONFIGURATION_ONLY replicas.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 [DEBUG] Promote info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Promoting replica to PRIMARY role...
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Replica is PRIMARY (1)
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Replica is now PRIMARY
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Monitor Caller is: promote.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Replica is PRIMARY (1)
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Instance name is server2.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Current master is server2.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 62 seconds
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 DB_FAILOVER is ON.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: 2023/07/19 14:56:14 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: promote: PROMOTION_SCORE: 20
>Jul 19 14:56:14  ag(ag_cluster)[71103]:    INFO: ag_cluster promote : 0
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: mssql_validate
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: mssql_notify post-promote
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: notify: 2023/07/19 14:56:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-promote]; connection-timeout [30]; health-threshold [3]; action [post-promote]
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: notify: 2023/07/19 14:56:14 [DEBUG] AG Helper PostPromote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: notify: 2023/07/19 14:56:14 Replica is PRIMARY (1)
>Jul 19 14:56:14  ag(ag_cluster)[71374]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: mssql_validate
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: mssql_monitor
>Jul 19 14:56:14  IPaddr2(virtualip)[71461]:    INFO: Adding inet address 192.168.0.200/24 with broadcast address 192.168.0.255 to device eth0
>Jul 19 14:56:14  IPaddr2(virtualip)[71461]:    INFO: Bringing device eth0 up
>Jul 19 14:56:14  IPaddr2(virtualip)[71461]:    INFO: /usr/libexec/heartbeat/send_arp  -i 200 -r 5 -p /run/resource-agents/send_arp-192.168.0.200 eth0 192.168.0.200 auto not_used not_used
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:14 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:14 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:14 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:56:14  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:14 Connected to the instance at localhost:1433
>Jul 19 14:56:18  IPaddr2(virtualip)[71461]:    INFO: ARPING 192.168.0.200 from 192.168.0.200 eth0
>Sent 5 probes (5 broadcast(s))
>Received 0 response(s)
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 Monitor Caller is: monitor.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 Replica is PRIMARY (1)
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 Instance name is server2.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 Current master is server2.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 DB_FAILOVER is ON.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: 2023/07/19 14:56:19 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:56:19  ag(ag_cluster)[71449]:    INFO: ag_cluster monitor : 8
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: mssql_validate
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: mssql_notify pre-stop
>Jul 19 14:56:27  ag(ag_cluster)[71801]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: mssql_validate
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: mssql_notify pre-stop
>Jul 19 14:56:27  ag(ag_cluster)[71846]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: mssql_validate
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: mssql_notify post-stop
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: notify: 2023/07/19 14:56:27 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-post-stop]; connection-timeout [30]; health-threshold [3]; action [post-stop]
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: notify: 2023/07/19 14:56:27 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: notify: 2023/07/19 14:56:27 Replica is PRIMARY (1)
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: notify: 2023/07/19 14:56:27 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: notify: 2023/07/19 14:56:27 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:27  ag(ag_cluster)[71891]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:27  ag(ag_cluster)[71978]:    INFO: mssql_validate
>Jul 19 14:56:27  ag(ag_cluster)[71978]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:27  ag(ag_cluster)[71978]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:27  ag(ag_cluster)[71978]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:27  ag(ag_cluster)[71978]:    INFO: mssql_notify pre-start
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 Replica is PRIMARY (1)
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: 2023/07/19 14:56:28 Sequence number is 3853:8 (61929133441032)
>Jul 19 14:56:28  ag(ag_cluster)[71978]:    INFO: notify: SEQUENCE_NUMBER: 61929133441032
>Jul 19 14:56:33  ag(ag_cluster)[71978]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: mssql_validate
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: mssql_monitor
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:33 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:33 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:33 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:56:33  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:33 Connected to the instance at localhost:1433
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 Monitor Caller is: monitor.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 Replica is PRIMARY (1)
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 Instance name is server2.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 Current master is server2.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 [DEBUG] Lease is in the process of being renewed for AVAILABILITY GROUP ag1 for 73 seconds
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 WARNING (Ignore this if External Lease is not used) : The AG does not allow updating external write lease. Not updating the external write lease. Recreate the AG with the WRITE_LEASE_VALIDITY option in the CREATE AVAILABILITY GROUP DDL.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 DB_FAILOVER is ON.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: 2023/07/19 14:56:38 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: monitor: PROMOTION_SCORE: 20
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:56:38  ag(ag_cluster)[72089]:    INFO: ag_cluster monitor : 8
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: mssql_validate
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: mssql_notify pre-start
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 Replica is PRIMARY (1)
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: 2023/07/19 14:56:38 Sequence number is 3853:B (61929133441035)
>Jul 19 14:56:38  ag(ag_cluster)[72316]:    INFO: notify: SEQUENCE_NUMBER: 61929133441035
>Jul 19 14:56:43  ag(ag_cluster)[72316]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: mssql_validate
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: mssql_notify pre-start
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-pre-start]; connection-timeout [30]; health-threshold [3]; action [pre-start]
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 Replica is PRIMARY (1)
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 AG has 3 SYNCHRONOUS_COMMIT replicas.
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 Setting REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT to 1...
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 Replica is SYNCHRONOUS_COMMIT (1).
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: 2023/07/19 14:56:43 Sequence number is 3853:B (61929133441035)
>Jul 19 14:56:43  ag(ag_cluster)[72426]:    INFO: notify: SEQUENCE_NUMBER: 61929133441035
>Jul 19 14:56:48  ag(ag_cluster)[72426]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: mssql_validate
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: OCF_RESKEY_CRM_meta_interval value: 11000
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: Resource agent invoked with: monitor
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: mssql_monitor
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:49 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-monitor]; connection-timeout [30]; health-threshold [3]; action [monitor]
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:49 ag-helper invoked with required-synchronized-secondaries-to-commit [-1]; current-master [server1
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: monitor: server2]; disable-primary-on-quorum-timeout-after [60]; primary-write-lease-duration [73]; monitor-interval-timeout [60]
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:49 From RetryExecuteWithTimeout - Attempt 1 to connect to the instance at localhost:1433
>Jul 19 14:56:49  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:49 Connected to the instance at localhost:1433
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Monitor Caller is: monitor.
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 [DEBUG] AG Helper Monitor Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Replica is PRIMARY (1)
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Offlining replica...
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Replica is RESOLVING (0)
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Instance name is server2.
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Current master is server1
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: server2.
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Setting the role to Secondary.
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Setting replica to SECONDARY role...
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: 2023/07/19 14:56:54 Could not set replica to SECONDARY role. Failover Failed.
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: monitor: PROMOTION_SCORE: -INFINITY
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: lease_expiry after monitor update: 
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: lease information from all replicas after monitor update: name="ag_cluster-lease-expiry" host="server1" value=""
>name="ag_cluster-lease-expiry" host="server3" value=""
>name="ag_cluster-lease-expiry" host="server2" value=""
>Jul 19 14:56:54  ag(ag_cluster)[72588]:    INFO: ag_cluster monitor : 0
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: mssql_validate
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: mssql_notify post-start
>Jul 19 14:56:54  ag(ag_cluster)[72827]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: mssql_validate
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:55  IPaddr2(virtualip)[72873]:    INFO: IP status = ok, IP_CIP=
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: mssql_notify pre-demote
>Jul 19 14:56:55  ag(ag_cluster)[72874]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: mssql_validate
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: Resource agent invoked with: demote
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: mssql_demote
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: demote: 2023/07/19 14:56:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-demote]; connection-timeout [30]; health-threshold [3]; action [demote]
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: demote: 2023/07/19 14:56:55 [DEBUG] AG Helper Demote Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: demote: 2023/07/19 14:56:55 Replica is RESOLVING (0)
>Jul 19 14:56:55  ag(ag_cluster)[72972]:    INFO: ag_cluster demote : 0
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: mssql_validate
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: mssql_notify post-demote
>Jul 19 14:56:55  ag(ag_cluster)[73064]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: mssql_validate
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 60000
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: Resource agent invoked with: notify
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: mssql_notify pre-stop
>Jul 19 14:56:55  ag(ag_cluster)[73109]:    INFO: ag_cluster notify : 0
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: mssql_validate
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: OCF_RESKEY_CRM_meta_interval value: 0
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: OCF_RESKEY_CRM_meta_timeout value: 10000
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: Resource agent invoked with: stop
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: mssql_stop
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: stop: 2023/07/19 14:56:55 ag-helper invoked with hostname [localhost]; port [1433]; ag-name [ag1]; credentials-file [/var/opt/mssql/secrets/passwd]; application-name [monitor-ag_cluster-stop]; connection-timeout [30]; health-threshold [3]; action [stop]
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: stop: 2023/07/19 14:56:55 [DEBUG] AG Helper Stop Role info: AVAILABILITY GROUP ag1 on instance server2
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: stop: 2023/07/19 14:56:55 Replica is RESOLVING (0)
>Jul 19 14:56:55  ag(ag_cluster)[73154]:    INFO: ag_cluster stop : 0

Actions: View

Attachments on bug 2221772: 1976118 | 1976119 | 1976120 | 1976684 | 1976685 | 1976686