Bug 2247183

Description of problem:

Currently, there is no easy way for an administrator to check the sync replication status between zones.  

Goal:

multisite sync observability: tracking sync deltas over time

Our proposed feature will increase the observability of the RGW multisite sync operations. It will provide administrators with real-time information about the replication health between zones. This will enable the admin to assess if the pending sync replication work is converging as expected or diverging. If it diverges and increases beyond a certain threshold, an alert can be configured in the alert manager to fire a warning.

To present this information to the user, we will use Prometheus to gather data and create a Grafana dashboard with data points representing the oldest incremental change not applied from the sync status command to populate the graph over time. 

The Grafana dashboard will display a slope to help us assess if the pending sync deltas are reducing or increasing over time. The ‘deltas’ will be sent from all the zones replicated in the zone group to Prometheus via the node-exporter.

Ideally, further down the line, we will be able to do similar work to have per-bucket granularity sync information in Prometheus so we can adhere to the bucket sync policy granularity that provides the user with a way to enable/disable bucket sync through the S3 API.