Description of problem:

The bug fix for replication of encrypted multipart objects in 6.1z1 (https://bugzilla.redhat.com/show_bug.cgi?id=2162337) also enabled a new feature: compression and encryption can be combined on the same object. That bug fix also resolved some issues with the replication of compressed+encrypted objects.

But in a multisite configuration where only zone has upgraded to 6.1z1, uploads to that zone can be both compressed and encrypted, but their replication to older zones without these fixes will corrupt the data. So while fixing one cause of data corruption, we introduced a new one.

Version-Release number of selected component (if applicable):

This effects the interoperability of multisite configurations with some zones on 6.1z1 and other zones on older releases.

How reproducible:

Whenever objects are uploaded with both compression and encryption to an upgraded 6.1z1 zone while older zones in the zonegroup are replicating from it. Customers generally upgrade one cluster at a time, so this will be a common scenario.

Steps to Reproduce:

1. Deploy two zones in the same zonegroup on RHCS 6.1 and verify that replication is working

2. Enable compression on each zone:

$ radosgw-admin zone placement modify \
      --rgw-zone <name> \
      --placement-id default-placement \
      --storage-class STANDARD \
      --compression zlib
$ radosgw-admin period update --commit

3. Configure default encryption on each zone: rgw crypt default encryption key = 4YSmvJtBv0aZ7geVgAsdpRnLBEwWSWlMIGnRS8a9TSA=

4. Upgrade one zone to 6.1z1

5. On the upgraded zone, create a bucket and upload a non-empty object

6. After waiting for replication, download that object from the other zone


Actual results:

A checksum of the replicated object data does not match the original upload. For example:

$ s3cmd -c ../work/c2.s3cfg get s3://testbucket/6m 6m.c2
download: 's3://testbucket/6m' -> '6m.c2'  [1 of 1]    
 6291456 of 6291456   100% in    0s   176.78 MB/s  done
WARNING: MD5 signatures do not match: computed=4101694a589baca05b76afb00a53206c, received=cca06bdd97b45abef3ac0f28f182ab69

Expected results:

Objects are replicated correctly even if zone versions don't match.


Additional info:

https://github.com/ceph/ceph/pull/52300 added a 'compress-encrypted' zone feature that can't be enabled until all zones upgrade to a supported version. This prevents objects from being both compressed and encrypted until we can guarantee that they will replicate correctly. This issue was raised in https://bugzilla.redhat.com/show_bug.cgi?id=2162337#c49 but it wasn't included in the backport.