Description of problem:
In a scaled testbed of pool_1 having 100 images in group_1
(4 images of size 2T,1T,500G,250G and rest with 50G)
Once all images are mirrored to secondary site:
[ceph: root@tala001 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0 | grep "syncing" | wc
      0       0       0
[ceph: root@tala001 /]#

performed demote on site-a:
[ceph: root@tala001 /]# rbd mirror group demote --pool pool_1 --group group_1 --debug_rbd 0
2025-04-22T16:50:44.485+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.486+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.491+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.492+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.493+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.495+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.498+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.505+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.505+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.510+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.511+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.532+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.533+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.534+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.620+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.621+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.621+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.622+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.627+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.479+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.481+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.485+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.483+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.483+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.484+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.858+0000 7fb48bfff640 -1 librbd::image_watcher::NotifyLockOwner: 0x7fb47824d6d0 handle_notify: duplicate lock owners detected
2025-04-22T16:50:53.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:53.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:53.730+0000 7fb48bfff640 -1 librbd::image_watcher::NotifyLockOwner: 0x7fb478210830 handle_notify: duplicate lock owners detected
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.497+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.500+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.500+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.501+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55931af169c0 handle_remove_group_snapshot: failed to remove image snapshot metadata: (22) Invalid argument
Group demoted to non-primary
[ceph: root@tala001 /]#


After demote, mirror group status is as below:
[ceph: root@tala001 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0
group_1:
  global_id:   2547a366-7ccb-4a15-955d-5092668a1e58
  state:       up+unknown
  description: remote group is non-primary
  service:     tala001.fkzuhy on tala001
  last_update: 2025-04-22 16:51:31
  images:
  peer_sites:
    name: site-b
    state: up+replaying
    description: replaying, {"last_snapshot_bytes":323308605440,"last_snapshot_complete_seconds":2331,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639}
    last_update: 2025-04-22 16:51:12
    images:
      image:       5/03a2f7e4-2491-4e18-8e0f-2aafb08f3430
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606756352.0,"last_snapshot_bytes":3213512704,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/05336a9b-c268-43e8-a9cf-92788c1de095
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611026432.0,"last_snapshot_bytes":3222052864,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/08717b95-431d-4d35-af02-416bfa97c928
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613088768.0,"last_snapshot_bytes":3226177536,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/0c76a355-cd61-4a5c-a98a-6aba6465ace6
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608744960.0,"last_snapshot_bytes":3217489920,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":155269820018086016,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/0cd9d0d1-94a1-471f-adda-a3e860ba428a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605691392.0,"last_snapshot_bytes":3211382784,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/10a810a2-eba8-4a29-9208-c5c120fb686a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609304064.0,"last_snapshot_bytes":3218608128,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/115fedb0-0a90-4f51-bff6-4397e338bf71
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612310528.0,"last_snapshot_bytes":3224621056,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1384964197751298560,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/150c627d-b38e-486c-85b6-f6f51ea6c398
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613582336.0,"last_snapshot_bytes":3227164672,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":660026986906979328,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1532090f-12aa-48d3-9b1f-8f0ec67cdb4b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608257536.0,"last_snapshot_bytes":3216515072,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/16e710d5-6680-4a74-9dce-742ebc51bc46
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1593745408.0,"last_snapshot_bytes":3187490816,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1b5d6d62-153c-485b-9bcf-d2fe81100172
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609101312.0,"last_snapshot_bytes":3218202624,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1c624ac0-e716-4338-a1df-09db7a43121e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605339136.0,"last_snapshot_bytes":3210678272,"last_snapshot_sync_seconds":69,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":250722875931232576,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2264ecdd-51d2-4ee1-b2f6-92fbaae6d93d
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605984256.0,"last_snapshot_bytes":3211968512,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/23a130be-4270-4325-af1d-6f1e15539c47
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601742848.0,"last_snapshot_bytes":3203485696,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":132296791322344208,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/286c603c-69c4-4f2d-86f7-b5c01e457074
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614893056.0,"last_snapshot_bytes":3229786112,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2cd905e3-6de3-491e-91e5-27df7cd0b369
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612687360.0,"last_snapshot_bytes":3225374720,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":165902741793355168,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2ce50b0c-97da-4f80-8175-6ad6fd7c7897
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601708032.0,"last_snapshot_bytes":3203416064,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/330c30ed-9307-43ea-9885-133e6bf1a3cc
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602560000.0,"last_snapshot_bytes":3205120000,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/37761a54-a32b-4437-bdc0-407512b475e8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613324288.0,"last_snapshot_bytes":3226648576,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/39735a9e-ed1f-4cfe-9df6-33ca99cbb224
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612847104.0,"last_snapshot_bytes":3225694208,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":91750007485143184,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3cf2f3b0-7d66-4b8e-aaba-56b782b2fead
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612148736.0,"last_snapshot_bytes":3224297472,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3ec38940-909e-4bca-b350-874434c6d4ad
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606060032.0,"last_snapshot_bytes":3212120064,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":645424590676277248,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3f18600a-9504-402f-9911-a91d44e8a4f2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614309376.0,"last_snapshot_bytes":3228618752,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":433337873471635456,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3fe78d62-5ac3-4279-acb9-eb1a6f289367
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615511552.0,"last_snapshot_bytes":3231023104,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/44ffeb91-addf-483f-8fdc-00379686d4c5
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609265152.0,"last_snapshot_bytes":3218530304,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/45bc76c8-8fc5-45cc-8d63-f0e70aaf5d4e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1604927488.0,"last_snapshot_bytes":3209854976,"last_snapshot_sync_seconds":90,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":270318081310252224,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/47bb028b-b58f-42ae-a55c-d0feb40bb534
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615534080.0,"last_snapshot_bytes":3231068160,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4b427273-8a5a-4617-bceb-f3c9742d45a8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611800576.0,"last_snapshot_bytes":3223601152,"last_snapshot_sync_seconds":71,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4b6f37fd-f11e-4220-a51f-be81aef343f6
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614286848.0,"last_snapshot_bytes":3228573696,"last_snapshot_sync_seconds":58,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":147517217415381344,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4dd53645-f998-4558-b914-5b438a48688e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611429888.0,"last_snapshot_bytes":3222859776,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":64381755058204120,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/51fa23f1-fbba-43c6-b36a-574014cb6fa3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608507392.0,"last_snapshot_bytes":3217014784,"last_snapshot_sync_seconds":73,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/5422c061-670e-4bfa-8e02-b3661d9bbcc0
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611171840.0,"last_snapshot_bytes":3222343680,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":256293717075412800,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/54c507b3-4267-47fa-aa85-fb6ac178d629
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609744384.0,"last_snapshot_bytes":3219488768,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/559ad0de-3e7b-4318-86ef-e9302be8c950
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1839300608.0,"last_snapshot_bytes":3678601216,"last_snapshot_sync_seconds":523,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":252791550683933312,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/57d3b28b-2727-4d3d-ac2f-15406d51b9a4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612431360.0,"last_snapshot_bytes":3224862720,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/597f0753-00f1-461d-81d1-1e0a38bfa665
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612695552.0,"last_snapshot_bytes":3225391104,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/5ec51ef1-1636-4015-aabd-921e0713c5ac
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611882496.0,"last_snapshot_bytes":3223764992,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":87702075760124800,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/63cac315-20ee-47eb-80f1-49fefdeefd7a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607634944.0,"last_snapshot_bytes":3215269888,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/63f621a2-ea2a-4170-88f5-dff1d7e7db9e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611069440.0,"last_snapshot_bytes":3222138880,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1581597841459436288,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/64726362-3107-48a4-9ff7-ff390e17b288
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1618563072.0,"last_snapshot_bytes":3237126144,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":41256234188446848,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/64cdc583-6a3a-4cfc-a27f-2a0d0744181f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1619521536.0,"last_snapshot_bytes":3239043072,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":86676536227883952,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/67f7cd94-907d-4c65-803f-ac7c57354475
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603012608.0,"last_snapshot_bytes":3206025216,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":55748070659398120,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6c7c8a9c-d26a-446e-989a-cbc5e66c3e19
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1729679360.0,"last_snapshot_bytes":3459358720,"last_snapshot_sync_seconds":469,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6d6dd610-00a8-408a-8385-c4f6403b4983
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606768640.0,"last_snapshot_bytes":3213537280,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":690101876103839744,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6e099078-79ec-468a-82f7-93ecf403208f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605029888.0,"last_snapshot_bytes":3210059776,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":229785029268751456,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6ed95678-36b2-4945-98f2-2acca0a51c9f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603235840.0,"last_snapshot_bytes":3206471680,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/74a851c5-1954-428a-9739-ff526044bc4a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1975875584.0,"last_snapshot_bytes":3951751168,"last_snapshot_sync_seconds":507,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/78972d69-0072-4361-bcce-58d3b7ae14dd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1618118656.0,"last_snapshot_bytes":3236237312,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":2779906683426989568,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/7eeaa4b0-bf61-4a0f-ac6e-17d5379ab5b9
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601267712.0,"last_snapshot_bytes":3202535424,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":53729628556099584,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/7eee4b5b-9746-4060-bdd7-7693a312cbe3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614358528.0,"last_snapshot_bytes":3228717056,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":144450355870389600,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/837dc7ee-99e1-4b95-b043-cb74f8be8906
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609488384.0,"last_snapshot_bytes":3218976768,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/83a9cfae-515d-416c-b3ea-f5b88f9ba66c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608910848.0,"last_snapshot_bytes":3217821696,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/85d31c4e-7d9f-4781-af14-779fe312e199
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605931008.0,"last_snapshot_bytes":3211862016,"last_snapshot_sync_seconds":77,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/892b5284-88ae-460f-85d0-aba4d79db97a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605269504.0,"last_snapshot_bytes":3210539008,"last_snapshot_sync_seconds":73,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":74941087184197184,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/8ba06e89-c0cf-4510-897e-bdca52533269
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609390080.0,"last_snapshot_bytes":3218780160,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":152757519560371808,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/913f7c0f-aee4-4860-9c1e-bca4a0d499cd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612083200.0,"last_snapshot_bytes":3224166400,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":100345574238130832,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/91867b7d-3716-4980-b2a8-c4716dd0efa3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606940672.0,"last_snapshot_bytes":3213881344,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":179266432022136704,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/91dab002-c406-4db2-84c5-4d79af675eb8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1598590976.0,"last_snapshot_bytes":3197181952,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":653894853485782912,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/92392c1e-2a9a-4234-ad89-3d62245558b1
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609115648.0,"last_snapshot_bytes":3218231296,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":111469340058739488,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/955c9ef2-be5c-47e6-b969-04a53dfdd422
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607579648.0,"last_snapshot_bytes":3215159296,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":242263228557024288,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9a300be5-8fa6-472b-9628-bad6d97a9778
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614934016.0,"last_snapshot_bytes":3229868032,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":730114608833467392,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9a908d7e-b237-4cd1-a4fe-b9d27cb892e8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603713024.0,"last_snapshot_bytes":3207426048,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9ba827d8-6f36-4544-8f49-b7a2389b4f2c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611274240.0,"last_snapshot_bytes":3222548480,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a0b03fc2-3a60-455e-ada8-f54db5816bd8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601839104.0,"last_snapshot_bytes":3203678208,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":237236088452894592,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a5cdf720-8eac-481d-a5cc-e811223cccd2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614075904.0,"last_snapshot_bytes":3228151808,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":159365591286014624,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a7104d6a-72f2-41a8-81c6-ab72c65afc25
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605855232.0,"last_snapshot_bytes":3211710464,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/af2b9348-635b-4dbf-8ad8-a9cd69dbf50f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613111296.0,"last_snapshot_bytes":3226222592,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":78064904350740000,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b08b063a-3b9c-4cc0-b0ee-6547e4a617e4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607069696.0,"last_snapshot_bytes":3214139392,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3562483502819438592,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b13d5c5d-e5e9-4f7a-8dc5-7299fe83bec5
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602246656.0,"last_snapshot_bytes":3204493312,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":291284528577581440,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b157b397-6c71-4ab5-8754-40eb1eb7e39c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1617684480.0,"last_snapshot_bytes":3235368960,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":82713118295794832,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b19e8a0e-7db8-4e1f-9dff-0bbf7f91f1bd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613621248.0,"last_snapshot_bytes":3227242496,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3080200217018091520,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b23f8fc1-70ce-47d3-9f93-64596bffe131
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605611520.0,"last_snapshot_bytes":3211223040,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b3ea7a69-81f0-4e29-869d-c455a9a9c96e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607591936.0,"last_snapshot_bytes":3215183872,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b8e9f42a-d502-413d-9feb-c1eeb1c886c2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605013504.0,"last_snapshot_bytes":3210027008,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":306376911525260672,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b9b25abb-0ca3-467c-9617-6cdef9d75d77
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612355584.0,"last_snapshot_bytes":3224711168,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/ba24354e-c247-4dfa-8a9c-f87edaa88b70
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609871360.0,"last_snapshot_bytes":3219742720,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":345717242098352128,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/bb9f01a6-9a64-4190-9fa1-30bf93acca8c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609314304.0,"last_snapshot_bytes":3218628608,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3455976152332500992,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/bcd3137c-50ac-400d-a121-144d65ee4ffd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602732032.0,"last_snapshot_bytes":3205464064,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":809844901375485440,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/beb1ac3f-7b5e-40a3-b59a-9234864a27cc
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605226496.0,"last_snapshot_bytes":3210452992,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/cce2ed20-c818-43ba-81e3-62b5af32a609
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607481344.0,"last_snapshot_bytes":3214962688,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/cd969442-b31c-42f0-afbc-f69fcb162b03
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614614528.0,"last_snapshot_bytes":3229229056,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/d0598ba4-240d-4c52-9d8b-d8ebff088ea4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605834752.0,"last_snapshot_bytes":3211669504,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/d4d5711c-3f2e-4958-bd8a-96fce8ec6d11
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601415168.0,"last_snapshot_bytes":3202830336,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":687802577387834624,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/dbf149f8-1e46-4a0e-bafd-08873690eb1b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615169536.0,"last_snapshot_bytes":3230339072,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":163225890226246944,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/df759833-8b06-4f02-b9e0-1803eb94388c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603840000.0,"last_snapshot_bytes":3207680000,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1197989625742024448,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e2c64d2e-f841-462e-aab0-2bb04d8a5a8c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605500928.0,"last_snapshot_bytes":3211001856,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":63262146600528904,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e31a8084-e16b-41f0-aa38-01a9fd18d36a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603690496.0,"last_snapshot_bytes":3207380992,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":61773975185883576,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e31f8429-f1d7-4a01-ab45-9d40b9338952
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611022336.0,"last_snapshot_bytes":3222044672,"last_snapshot_sync_seconds":90,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e589ef63-8883-4fa4-b6d7-be86745ff11e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1600479232.0,"last_snapshot_bytes":3200958464,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/ef65ae60-007d-435d-b7a7-ce072fa59d86
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609867264.0,"last_snapshot_bytes":3219734528,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1382865449956199680,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f1c86f50-bc97-4826-8612-fa4f553fe599
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1610668032.0,"last_snapshot_bytes":3221336064,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f55e2b9d-0571-4ca7-bbaf-d2ae18bd4528
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609492480.0,"last_snapshot_bytes":3218984960,"last_snapshot_sync_seconds":69,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":114200806438954000,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f706c9a1-3de5-40ad-b2c0-2279ea82be9c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607471104.0,"last_snapshot_bytes":3214942208,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f8a69623-502e-4904-8b19-28245fc62116
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1686689792.0,"last_snapshot_bytes":3373379584,"last_snapshot_sync_seconds":395,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1525111051607798528,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f8dc6704-2918-48fc-82cd-a24109ed4f7b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605488640.0,"last_snapshot_bytes":3210977280,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":154955532649427360,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fba22c6a-b901-4ea4-986d-55ae82b69b40
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1610201088.0,"last_snapshot_bytes":3220402176,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":350165114579423680,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fbc51c47-2647-410c-9596-6bf7765f19a9
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1595959296.0,"last_snapshot_bytes":3191918592,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":90789311020757392,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fc7d3cf8-02bf-48cc-b3c9-3c3890d9edb8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602693120.0,"last_snapshot_bytes":3205386240,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":72841423660573584,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fcb1605a-0322-41bd-937d-a3bebce1af95
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612584960.0,"last_snapshot_bytes":3225169920,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fcd706f1-fe49-4038-934e-393c8d29d5f0
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1599858688.0,"last_snapshot_bytes":3199717376,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":808393028609584384,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}
[ceph: root@tala001 /]# 


>> Trying promote on site-b, gives below error:
[ceph: root@tala005 /]# rbd mirror group promote --pool pool_1 --group group_1 --debug_rbd 0
rbd: error promoting group to primary
2025-04-22T16:59:08.686+0000 7f07bf48f380 -1 librbd::api::Mirror: group_promote: group group_1 is still primary within a remote cluster
[ceph: root@tala005 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0 | grep '"syncing_percent":0' | wc
    100     300   33751
[ceph: root@tala005 /]# date
Tue Apr 22 17:02:32 UTC 2025
[ceph: root@tala005 /]# rbd mirror group promote --pool pool_1 --group group_1 --debug_rbd 0
rbd: error promoting group to primary
2025-04-22T17:03:09.431+0000 7f41be707380 -1 librbd::api::Mirror: group_promote: group group_1 is still primary within a remote cluster
[ceph: root@tala005 /]#
[ceph: root@tala005 /]# rbd group info --pool pool_1 --group group_1 --debug_rbd 0
rbd group 'group_1':
        id: 1251fc3c85479
        mirroring state: enabled
        mirroring mode: snapshot
        mirroring global id: 2547a366-7ccb-4a15-955d-5092668a1e58
        mirroring primary: false
[ceph: root@tala005 /]#

As the cluster is non-primary, it should not give error as "group group_1 is still primary within a remote cluster"
Also, 2 more observations noticed are :
1) Group status stuck in syncing_percent: 0 for all 100 images for long
2) Seeing some garbage value for "seconds_until_synced":1525111051607798528 for some of the images in mirror group status.


group snap list on both the cluster is as below:
site-a:
[ceph: root@tala001 /]# rbd group snap ls --pool pool_1 --group group_1 --debug_rbd 0
ID             NAME                                                                STATE       NAMESPACE                         
15cfc3cc18b7c  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.15cfc3cc18b7c    complete  mirror (primary peer_uuids:[])    
18bbb90e19bf5  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bbb90e19bf5    complete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bc72542d363  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bc72542d363  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bcdc825d1a2  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bcdc825d1a2  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bd32e5ee578  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bd32e5ee578  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18cbde4a21061  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18cbde4a21061    complete  mirror (demoted peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
[ceph: root@tala001 /]#

site-b:
[ceph: root@tala005 /]# rbd group snap ls --pool pool_1 --group group_1 --debug_rbd 0
ID             NAME                                                                    STATE     NAMESPACE                                                         
18bbb90e19bf5  .mirror.non-primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bbb90e19bf5  complete  mirror (non-primary peer_uuids:[] 0d5fe4a4-a87b-4451-b377-c4006436a1d1:18bbb90e19bf5)
[ceph: root@tala005 /]#
Version-Release number of selected component (if applicable):
[ceph: root@tala001 /]# ceph version
ceph version 19.2.1-151.el9cp (642684c3e6e5556e1132f363e49d5e26b605e8ef) squid (stable)
[ceph: root@tala001 /]#


Actual results: Promote is failing with scale


Expected results: Promote should succeed


Additional info: NA