Description of problem:

Hi,

When testing the RGW Multisite(16.2.8-85.el8cp) sync policy, modifications/updates to objects take place during the zonegroup or bucket replication is set to "Allowed" or "Disabled" don't reconcile/sync once we configure the replication of the zonegroup or bucket to "enabled".

So for the simplest example at the zonegroup level, I create zone1/zone2 symmetrical replication:

$ radosgw-admin sync group create --group-id=us --status=enabled --rgw-zonegroup=multizone  --rgw-zone=zone2
$ radosgw-admin sync group flow create --group-id=us --flow-id=us-1-to-2 --flow-type=symmetrical --zones=zone1,zone2 --rgw-zonegroup=multizone --rgw-realm=multisite  --rgw-zone=zone2
$ radosgw-admin sync group pipe create --group-id=us --pipe-id=ussynctoall --source-zones='*' --source-bucket='*' --dest-zones='*' --dest-bucket='*'  --rgw-zone=zone2
$ radosgw-admin period update --commit

I then upload objects to bucket1 in zone2, that get replicated ok. 

$ aws s3  --profile ms  --endpoint http://zone2:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2

$ aws s3  --profile ms  --endpoint http://zone1:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2

I then set the sync group to allowed:

$ radosgw-admin sync group modify --group-id=us --status=allowed
$ radosgw-admin period update --commit

Upload objects to bucket1 in zone2, and they don't get replicated, so expected behaviour.

$ aws s3  --profile ms  --endpoint http://zone2:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2
2022-12-14 17:23:52       4066 hosts3
2022-12-14 17:27:02       4066 hosts4
2022-12-14 17:27:15       4066 hosts5

$ aws s3  --profile ms  --endpoint http://zone1:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2


I then set the zonegroup replication back to 'Enabled':

$ radosgw-admin sync group modify --group-id=us --status=enabled
$ radosgw-admin period update --commit 

And copy one new object host6:

$ aws s3  --profile ms  --endpoint http://zone2:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2
2022-12-14 17:23:52       4066 hosts3
2022-12-14 17:27:02       4066 hosts4
2022-12-14 17:27:15       4066 hosts5
2022-12-14 17:29:15       4066 hosts6

host6 file gets replicated ok but files hosts3-5 that were uploaded during the time the zone replication was in 'Allowed' state don't get synced:

$ aws s3  --profile ms  --endpoint http://zone2:8080 ls s3://bucket1/
2022-12-14 15:20:22       4066 hosts
2022-12-15 14:55:54       4066 hosts1
2022-12-15 15:22:35       4066 hosts2
2022-12-14 17:29:15       4066 hosts6

From what I see in this tracker this is expected behaviour?
https://tracker.ceph.com/issues/57489

This behaviour is different from what customers expect. Getting an automated sync of all the modified objects during the 'Allowed' state would be a critical feature for customers to keep their buckets in sync between zones without manual intervention.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: