Bug 2361701 - [Scale] Consistency group mirroring promote failing with "group group_1 is still primary within a remote cluster"
Summary: [Scale] Consistency group mirroring promote failing with "group group_1 is st...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD-Mirror
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 8.1
Assignee: Ilya Dryomov
QA Contact: aarsharm
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-04-22 17:17 UTC by aarsharm
Modified: 2025-06-26 12:30 UTC (History)
3 users (show)

Fixed In Version: ceph-19.2.1-166.el9cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-06-26 12:30:51 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-11231 0 None None None 2025-04-22 17:19:27 UTC
Red Hat Product Errata RHSA-2025:9775 0 None None None 2025-06-26 12:30:54 UTC

Description aarsharm 2025-04-22 17:17:20 UTC
Description of problem:
In a scaled testbed of pool_1 having 100 images in group_1
(4 images of size 2T,1T,500G,250G and rest with 50G)
Once all images are mirrored to secondary site:
[ceph: root@tala001 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0 | grep "syncing" | wc
      0       0       0
[ceph: root@tala001 /]#

performed demote on site-a:
[ceph: root@tala001 /]# rbd mirror group demote --pool pool_1 --group group_1 --debug_rbd 0
2025-04-22T16:50:44.485+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.486+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.489+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.491+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.492+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.493+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.495+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.498+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.505+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.505+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.507+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.508+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.509+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.510+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.511+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.530+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.532+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.533+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.534+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.620+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.621+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.621+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.622+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:44.627+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.479+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.480+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.481+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:51.485+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.483+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.483+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.484+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:52.858+0000 7fb48bfff640 -1 librbd::image_watcher::NotifyLockOwner: 0x7fb47824d6d0 handle_notify: duplicate lock owners detected
2025-04-22T16:50:53.490+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:53.490+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:53.730+0000 7fb48bfff640 -1 librbd::image_watcher::NotifyLockOwner: 0x7fb478210830 handle_notify: duplicate lock owners detected
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.496+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:54.497+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.500+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.500+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.501+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.502+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.503+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.504+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb48bfff640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::Operations: update notification timed-out
2025-04-22T16:50:55.506+0000 7fb498a62640 -1 librbd::mirror::snapshot::GroupUnlinkPeerRequest: 0x55931af169c0 handle_remove_group_snapshot: failed to remove image snapshot metadata: (22) Invalid argument
Group demoted to non-primary
[ceph: root@tala001 /]#


After demote, mirror group status is as below:
[ceph: root@tala001 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0
group_1:
  global_id:   2547a366-7ccb-4a15-955d-5092668a1e58
  state:       up+unknown
  description: remote group is non-primary
  service:     tala001.fkzuhy on tala001
  last_update: 2025-04-22 16:51:31
  images:
  peer_sites:
    name: site-b
    state: up+replaying
    description: replaying, {"last_snapshot_bytes":323308605440,"last_snapshot_complete_seconds":2331,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639}
    last_update: 2025-04-22 16:51:12
    images:
      image:       5/03a2f7e4-2491-4e18-8e0f-2aafb08f3430
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606756352.0,"last_snapshot_bytes":3213512704,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/05336a9b-c268-43e8-a9cf-92788c1de095
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611026432.0,"last_snapshot_bytes":3222052864,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/08717b95-431d-4d35-af02-416bfa97c928
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613088768.0,"last_snapshot_bytes":3226177536,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/0c76a355-cd61-4a5c-a98a-6aba6465ace6
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608744960.0,"last_snapshot_bytes":3217489920,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":155269820018086016,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/0cd9d0d1-94a1-471f-adda-a3e860ba428a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605691392.0,"last_snapshot_bytes":3211382784,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/10a810a2-eba8-4a29-9208-c5c120fb686a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609304064.0,"last_snapshot_bytes":3218608128,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/115fedb0-0a90-4f51-bff6-4397e338bf71
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612310528.0,"last_snapshot_bytes":3224621056,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1384964197751298560,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/150c627d-b38e-486c-85b6-f6f51ea6c398
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613582336.0,"last_snapshot_bytes":3227164672,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":660026986906979328,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1532090f-12aa-48d3-9b1f-8f0ec67cdb4b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608257536.0,"last_snapshot_bytes":3216515072,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/16e710d5-6680-4a74-9dce-742ebc51bc46
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1593745408.0,"last_snapshot_bytes":3187490816,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1b5d6d62-153c-485b-9bcf-d2fe81100172
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609101312.0,"last_snapshot_bytes":3218202624,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/1c624ac0-e716-4338-a1df-09db7a43121e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605339136.0,"last_snapshot_bytes":3210678272,"last_snapshot_sync_seconds":69,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":250722875931232576,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2264ecdd-51d2-4ee1-b2f6-92fbaae6d93d
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605984256.0,"last_snapshot_bytes":3211968512,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/23a130be-4270-4325-af1d-6f1e15539c47
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601742848.0,"last_snapshot_bytes":3203485696,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":132296791322344208,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/286c603c-69c4-4f2d-86f7-b5c01e457074
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614893056.0,"last_snapshot_bytes":3229786112,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2cd905e3-6de3-491e-91e5-27df7cd0b369
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612687360.0,"last_snapshot_bytes":3225374720,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":165902741793355168,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/2ce50b0c-97da-4f80-8175-6ad6fd7c7897
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601708032.0,"last_snapshot_bytes":3203416064,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/330c30ed-9307-43ea-9885-133e6bf1a3cc
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602560000.0,"last_snapshot_bytes":3205120000,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/37761a54-a32b-4437-bdc0-407512b475e8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613324288.0,"last_snapshot_bytes":3226648576,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/39735a9e-ed1f-4cfe-9df6-33ca99cbb224
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612847104.0,"last_snapshot_bytes":3225694208,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":91750007485143184,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3cf2f3b0-7d66-4b8e-aaba-56b782b2fead
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612148736.0,"last_snapshot_bytes":3224297472,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3ec38940-909e-4bca-b350-874434c6d4ad
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606060032.0,"last_snapshot_bytes":3212120064,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":645424590676277248,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3f18600a-9504-402f-9911-a91d44e8a4f2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614309376.0,"last_snapshot_bytes":3228618752,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":433337873471635456,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/3fe78d62-5ac3-4279-acb9-eb1a6f289367
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615511552.0,"last_snapshot_bytes":3231023104,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/44ffeb91-addf-483f-8fdc-00379686d4c5
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609265152.0,"last_snapshot_bytes":3218530304,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/45bc76c8-8fc5-45cc-8d63-f0e70aaf5d4e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1604927488.0,"last_snapshot_bytes":3209854976,"last_snapshot_sync_seconds":90,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":270318081310252224,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/47bb028b-b58f-42ae-a55c-d0feb40bb534
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615534080.0,"last_snapshot_bytes":3231068160,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4b427273-8a5a-4617-bceb-f3c9742d45a8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611800576.0,"last_snapshot_bytes":3223601152,"last_snapshot_sync_seconds":71,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4b6f37fd-f11e-4220-a51f-be81aef343f6
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614286848.0,"last_snapshot_bytes":3228573696,"last_snapshot_sync_seconds":58,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":147517217415381344,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/4dd53645-f998-4558-b914-5b438a48688e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611429888.0,"last_snapshot_bytes":3222859776,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":64381755058204120,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/51fa23f1-fbba-43c6-b36a-574014cb6fa3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608507392.0,"last_snapshot_bytes":3217014784,"last_snapshot_sync_seconds":73,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/5422c061-670e-4bfa-8e02-b3661d9bbcc0
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611171840.0,"last_snapshot_bytes":3222343680,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":256293717075412800,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/54c507b3-4267-47fa-aa85-fb6ac178d629
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609744384.0,"last_snapshot_bytes":3219488768,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/559ad0de-3e7b-4318-86ef-e9302be8c950
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1839300608.0,"last_snapshot_bytes":3678601216,"last_snapshot_sync_seconds":523,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":252791550683933312,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/57d3b28b-2727-4d3d-ac2f-15406d51b9a4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612431360.0,"last_snapshot_bytes":3224862720,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/597f0753-00f1-461d-81d1-1e0a38bfa665
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612695552.0,"last_snapshot_bytes":3225391104,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/5ec51ef1-1636-4015-aabd-921e0713c5ac
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611882496.0,"last_snapshot_bytes":3223764992,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":87702075760124800,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/63cac315-20ee-47eb-80f1-49fefdeefd7a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607634944.0,"last_snapshot_bytes":3215269888,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/63f621a2-ea2a-4170-88f5-dff1d7e7db9e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611069440.0,"last_snapshot_bytes":3222138880,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1581597841459436288,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/64726362-3107-48a4-9ff7-ff390e17b288
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1618563072.0,"last_snapshot_bytes":3237126144,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":41256234188446848,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/64cdc583-6a3a-4cfc-a27f-2a0d0744181f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1619521536.0,"last_snapshot_bytes":3239043072,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":86676536227883952,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/67f7cd94-907d-4c65-803f-ac7c57354475
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603012608.0,"last_snapshot_bytes":3206025216,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":55748070659398120,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6c7c8a9c-d26a-446e-989a-cbc5e66c3e19
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1729679360.0,"last_snapshot_bytes":3459358720,"last_snapshot_sync_seconds":469,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6d6dd610-00a8-408a-8385-c4f6403b4983
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606768640.0,"last_snapshot_bytes":3213537280,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":690101876103839744,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6e099078-79ec-468a-82f7-93ecf403208f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605029888.0,"last_snapshot_bytes":3210059776,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":229785029268751456,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/6ed95678-36b2-4945-98f2-2acca0a51c9f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603235840.0,"last_snapshot_bytes":3206471680,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/74a851c5-1954-428a-9739-ff526044bc4a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1975875584.0,"last_snapshot_bytes":3951751168,"last_snapshot_sync_seconds":507,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/78972d69-0072-4361-bcce-58d3b7ae14dd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1618118656.0,"last_snapshot_bytes":3236237312,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":2779906683426989568,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/7eeaa4b0-bf61-4a0f-ac6e-17d5379ab5b9
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601267712.0,"last_snapshot_bytes":3202535424,"last_snapshot_sync_seconds":91,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":53729628556099584,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/7eee4b5b-9746-4060-bdd7-7693a312cbe3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614358528.0,"last_snapshot_bytes":3228717056,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":144450355870389600,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/837dc7ee-99e1-4b95-b043-cb74f8be8906
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609488384.0,"last_snapshot_bytes":3218976768,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/83a9cfae-515d-416c-b3ea-f5b88f9ba66c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1608910848.0,"last_snapshot_bytes":3217821696,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/85d31c4e-7d9f-4781-af14-779fe312e199
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605931008.0,"last_snapshot_bytes":3211862016,"last_snapshot_sync_seconds":77,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/892b5284-88ae-460f-85d0-aba4d79db97a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605269504.0,"last_snapshot_bytes":3210539008,"last_snapshot_sync_seconds":73,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":74941087184197184,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/8ba06e89-c0cf-4510-897e-bdca52533269
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609390080.0,"last_snapshot_bytes":3218780160,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":152757519560371808,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/913f7c0f-aee4-4860-9c1e-bca4a0d499cd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612083200.0,"last_snapshot_bytes":3224166400,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":100345574238130832,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/91867b7d-3716-4980-b2a8-c4716dd0efa3
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1606940672.0,"last_snapshot_bytes":3213881344,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":179266432022136704,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/91dab002-c406-4db2-84c5-4d79af675eb8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1598590976.0,"last_snapshot_bytes":3197181952,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":653894853485782912,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/92392c1e-2a9a-4234-ad89-3d62245558b1
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609115648.0,"last_snapshot_bytes":3218231296,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":111469340058739488,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/955c9ef2-be5c-47e6-b969-04a53dfdd422
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607579648.0,"last_snapshot_bytes":3215159296,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":242263228557024288,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9a300be5-8fa6-472b-9628-bad6d97a9778
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614934016.0,"last_snapshot_bytes":3229868032,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":730114608833467392,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9a908d7e-b237-4cd1-a4fe-b9d27cb892e8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603713024.0,"last_snapshot_bytes":3207426048,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/9ba827d8-6f36-4544-8f49-b7a2389b4f2c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611274240.0,"last_snapshot_bytes":3222548480,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a0b03fc2-3a60-455e-ada8-f54db5816bd8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601839104.0,"last_snapshot_bytes":3203678208,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":237236088452894592,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a5cdf720-8eac-481d-a5cc-e811223cccd2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614075904.0,"last_snapshot_bytes":3228151808,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":159365591286014624,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/a7104d6a-72f2-41a8-81c6-ab72c65afc25
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605855232.0,"last_snapshot_bytes":3211710464,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/af2b9348-635b-4dbf-8ad8-a9cd69dbf50f
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613111296.0,"last_snapshot_bytes":3226222592,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":78064904350740000,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b08b063a-3b9c-4cc0-b0ee-6547e4a617e4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607069696.0,"last_snapshot_bytes":3214139392,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3562483502819438592,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b13d5c5d-e5e9-4f7a-8dc5-7299fe83bec5
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602246656.0,"last_snapshot_bytes":3204493312,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":291284528577581440,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b157b397-6c71-4ab5-8754-40eb1eb7e39c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1617684480.0,"last_snapshot_bytes":3235368960,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":82713118295794832,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b19e8a0e-7db8-4e1f-9dff-0bbf7f91f1bd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1613621248.0,"last_snapshot_bytes":3227242496,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3080200217018091520,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b23f8fc1-70ce-47d3-9f93-64596bffe131
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605611520.0,"last_snapshot_bytes":3211223040,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b3ea7a69-81f0-4e29-869d-c455a9a9c96e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607591936.0,"last_snapshot_bytes":3215183872,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b8e9f42a-d502-413d-9feb-c1eeb1c886c2
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605013504.0,"last_snapshot_bytes":3210027008,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":306376911525260672,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/b9b25abb-0ca3-467c-9617-6cdef9d75d77
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612355584.0,"last_snapshot_bytes":3224711168,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/ba24354e-c247-4dfa-8a9c-f87edaa88b70
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609871360.0,"last_snapshot_bytes":3219742720,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":345717242098352128,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/bb9f01a6-9a64-4190-9fa1-30bf93acca8c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609314304.0,"last_snapshot_bytes":3218628608,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":3455976152332500992,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/bcd3137c-50ac-400d-a121-144d65ee4ffd
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602732032.0,"last_snapshot_bytes":3205464064,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":809844901375485440,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/beb1ac3f-7b5e-40a3-b59a-9234864a27cc
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605226496.0,"last_snapshot_bytes":3210452992,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/cce2ed20-c818-43ba-81e3-62b5af32a609
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607481344.0,"last_snapshot_bytes":3214962688,"last_snapshot_sync_seconds":57,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/cd969442-b31c-42f0-afbc-f69fcb162b03
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1614614528.0,"last_snapshot_bytes":3229229056,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/d0598ba4-240d-4c52-9d8b-d8ebff088ea4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605834752.0,"last_snapshot_bytes":3211669504,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/d4d5711c-3f2e-4958-bd8a-96fce8ec6d11
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1601415168.0,"last_snapshot_bytes":3202830336,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":687802577387834624,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/dbf149f8-1e46-4a0e-bafd-08873690eb1b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1615169536.0,"last_snapshot_bytes":3230339072,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":163225890226246944,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/df759833-8b06-4f02-b9e0-1803eb94388c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603840000.0,"last_snapshot_bytes":3207680000,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1197989625742024448,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e2c64d2e-f841-462e-aab0-2bb04d8a5a8c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605500928.0,"last_snapshot_bytes":3211001856,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":63262146600528904,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e31a8084-e16b-41f0-aa38-01a9fd18d36a
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1603690496.0,"last_snapshot_bytes":3207380992,"last_snapshot_sync_seconds":87,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":61773975185883576,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e31f8429-f1d7-4a01-ab45-9d40b9338952
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1611022336.0,"last_snapshot_bytes":3222044672,"last_snapshot_sync_seconds":90,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/e589ef63-8883-4fa4-b6d7-be86745ff11e
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1600479232.0,"last_snapshot_bytes":3200958464,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/ef65ae60-007d-435d-b7a7-ce072fa59d86
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609867264.0,"last_snapshot_bytes":3219734528,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1382865449956199680,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f1c86f50-bc97-4826-8612-fa4f553fe599
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1610668032.0,"last_snapshot_bytes":3221336064,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f55e2b9d-0571-4ca7-bbaf-d2ae18bd4528
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1609492480.0,"last_snapshot_bytes":3218984960,"last_snapshot_sync_seconds":69,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":114200806438954000,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f706c9a1-3de5-40ad-b2c0-2279ea82be9c
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1607471104.0,"last_snapshot_bytes":3214942208,"last_snapshot_sync_seconds":85,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f8a69623-502e-4904-8b19-28245fc62116
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1686689792.0,"last_snapshot_bytes":3373379584,"last_snapshot_sync_seconds":395,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":1525111051607798528,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/f8dc6704-2918-48fc-82cd-a24109ed4f7b
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1605488640.0,"last_snapshot_bytes":3210977280,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":154955532649427360,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fba22c6a-b901-4ea4-986d-55ae82b69b40
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1610201088.0,"last_snapshot_bytes":3220402176,"last_snapshot_sync_seconds":98,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":350165114579423680,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fbc51c47-2647-410c-9596-6bf7765f19a9
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1595959296.0,"last_snapshot_bytes":3191918592,"last_snapshot_sync_seconds":86,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":90789311020757392,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fc7d3cf8-02bf-48cc-b3c9-3c3890d9edb8
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1602693120.0,"last_snapshot_bytes":3205386240,"last_snapshot_sync_seconds":97,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":72841423660573584,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fcb1605a-0322-41bd-937d-a3bebce1af95
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1612584960.0,"last_snapshot_bytes":3225169920,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1745340639}

      image:       5/fcd706f1-fe49-4038-934e-393c8d29d5f0
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1599858688.0,"last_snapshot_bytes":3199717376,"last_snapshot_sync_seconds":99,"local_snapshot_timestamp":1745337935,"remote_snapshot_timestamp":1745340639,"replay_state":"syncing","seconds_until_synced":808393028609584384,"syncing_percent":0,"syncing_snapshot_timestamp":1745340639}
[ceph: root@tala001 /]# 


>> Trying promote on site-b, gives below error:
[ceph: root@tala005 /]# rbd mirror group promote --pool pool_1 --group group_1 --debug_rbd 0
rbd: error promoting group to primary
2025-04-22T16:59:08.686+0000 7f07bf48f380 -1 librbd::api::Mirror: group_promote: group group_1 is still primary within a remote cluster
[ceph: root@tala005 /]# rbd mirror group status --pool pool_1 --group group_1 --debug_rbd 0 | grep '"syncing_percent":0' | wc
    100     300   33751
[ceph: root@tala005 /]# date
Tue Apr 22 17:02:32 UTC 2025
[ceph: root@tala005 /]# rbd mirror group promote --pool pool_1 --group group_1 --debug_rbd 0
rbd: error promoting group to primary
2025-04-22T17:03:09.431+0000 7f41be707380 -1 librbd::api::Mirror: group_promote: group group_1 is still primary within a remote cluster
[ceph: root@tala005 /]#
[ceph: root@tala005 /]# rbd group info --pool pool_1 --group group_1 --debug_rbd 0
rbd group 'group_1':
        id: 1251fc3c85479
        mirroring state: enabled
        mirroring mode: snapshot
        mirroring global id: 2547a366-7ccb-4a15-955d-5092668a1e58
        mirroring primary: false
[ceph: root@tala005 /]#

As the cluster is non-primary, it should not give error as "group group_1 is still primary within a remote cluster"
Also, 2 more observations noticed are :
1) Group status stuck in syncing_percent: 0 for all 100 images for long
2) Seeing some garbage value for "seconds_until_synced":1525111051607798528 for some of the images in mirror group status.


group snap list on both the cluster is as below:
site-a:
[ceph: root@tala001 /]# rbd group snap ls --pool pool_1 --group group_1 --debug_rbd 0
ID             NAME                                                                STATE       NAMESPACE                         
15cfc3cc18b7c  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.15cfc3cc18b7c    complete  mirror (primary peer_uuids:[])    
18bbb90e19bf5  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bbb90e19bf5    complete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bc72542d363  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bc72542d363  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bcdc825d1a2  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bcdc825d1a2  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18bd32e5ee578  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bd32e5ee578  incomplete  mirror (primary peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
18cbde4a21061  .mirror.primary.2547a366-7ccb-4a15-955d-5092668a1e58.18cbde4a21061    complete  mirror (demoted peer_uuids:[b16d0c64-e83b-44f8-8bdb-6fc52214088d])
[ceph: root@tala001 /]#

site-b:
[ceph: root@tala005 /]# rbd group snap ls --pool pool_1 --group group_1 --debug_rbd 0
ID             NAME                                                                    STATE     NAMESPACE                                                         
18bbb90e19bf5  .mirror.non-primary.2547a366-7ccb-4a15-955d-5092668a1e58.18bbb90e19bf5  complete  mirror (non-primary peer_uuids:[] 0d5fe4a4-a87b-4451-b377-c4006436a1d1:18bbb90e19bf5)
[ceph: root@tala005 /]#
Version-Release number of selected component (if applicable):
[ceph: root@tala001 /]# ceph version
ceph version 19.2.1-151.el9cp (642684c3e6e5556e1132f363e49d5e26b605e8ef) squid (stable)
[ceph: root@tala001 /]#


Actual results: Promote is failing with scale


Expected results: Promote should succeed


Additional info: NA

Comment 10 errata-xmlrpc 2025-06-26 12:30:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775


Note You need to log in before you can comment on or make changes to this bug.