Bug 1699478 - rgw-multisite: log trimming does not make progress unless zones 'sync_from_all'
Summary: rgw-multisite: log trimming does not make progress unless zones 'sync_from_all'
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW-Multisite
Version: 3.2
Hardware: All
OS: Linux
Target Milestone: z2
: 3.2
Assignee: Casey Bodley
QA Contact: Tejas
Aron Gunn
: 1700042 (view as bug list)
Depends On:
Blocks: 1629656
TreeView+ depends on / blocked
Reported: 2019-04-12 21:05 UTC by Casey Bodley
Modified: 2019-11-11 08:02 UTC (History)
11 users (show)

Fixed In Version: RHEL: ceph-12.2.8-128.el7cp Ubuntu: ceph_12.2.8-111redhat1
Doc Type: Bug Fix
Doc Text:
.A multi-site Ceph Object Gateway is not trimming the data and bucket index logs Configuring zones for a multi-site Ceph Object Gateway without setting the `sync_from_all` option, was causing the data and bucket index logs not to be trimmed. With this release, the automated trimming process only consults the synchronization status of peer zones that are configured to synchronize. As result, this allows the data and bucket index logs to be trimmed properly.
Clone Of:
Last Closed: 2019-04-30 15:57:08 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 39283 None None None 2019-04-12 21:05:09 UTC
Ceph Project Bug Tracker 39487 None None None 2019-04-25 19:44:08 UTC
Red Hat Knowledge Base (Solution) 4079701 Troubleshoot None Ceph - RGW Multisite - bilogs and datalogs trimming do not make progress if 'sync_from_all' is not enabled. 2019-04-22 23:05:35 UTC
Red Hat Product Errata RHSA-2019:0911 None None None 2019-04-30 15:57:23 UTC

Description Casey Bodley 2019-04-12 21:05:10 UTC
Description of problem:

The trimming process for data logs and bucket index logs relies on querying the sync status of peer zones to determine how much of your log is safe to trim. Log entries are only safe to trim if all peer zones report a sync status marker that is larger than the given log entry.

The default zone configuration sets 'sync_from_all=true', meaning that it syncs data from each peer zone in its zonegroup. A zone can also be configured to only 'sync_from' a subset of the zonegroup. When such a zone does not sync from one of its peers, it will return emtpy markers when that peer requests its sync status. This will prevent the peer zone from making progress in trimming its data logs and bucket index logs.

Version-Release number of selected component (if applicable):

How reproducible:

Whenever a zone is configured to -not- sync from one of its peer zones.

Steps to Reproduce:
1. Create a multisite configuration with two zones 'a' and 'b'.

2. On the primary cluster, modify zone 'a' to not sync from zone 'b':
$ radosgw-admin zone modify --rgw-zone a --sync-from-all=0
$ radosgw-admin period update --commit

3. On the primary cluster, create a bucket 'bucket' and upload some objects.

4. Verify that the objects sync to the secondary zone and that sync status catches up.

5. Wait for at least rgw_sync_log_trim_interval (default 20min)

6. List the data log and bucket index log on each zone:
$ radosgw-admin datalog list
$ radosgw-admin bilog list --bucket bucket

Actual results:

The logs on zone 'a' are empty, but the logs on zone 'b' are not.

Expected results:

The logs on both zones are empty.

Additional info:

Comment 1 Vikhyat Umrao 2019-04-15 16:56:46 UTC
*** Bug 1700042 has been marked as a duplicate of this bug. ***

Comment 20 errata-xmlrpc 2019-04-30 15:57:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.