Description of problem:
Mirror status of some rbd images remain outdated for indefinite amount of time resulting in confusion regarding data sync.
Version-Release number of selected component (if applicable):
Faced issue on 2-3 images per 10 images created
Steps to Reproduce:
1. Configure two clusters and configure rbd-mirroring (We had established multi-secondary, One-way mirroring)
2. Create some images and wait for a while.
3. Check mirror image status and check 'last_update' and compare it with current system time.
Mirror image status of some images remains outdated for indefinite time
(status of xyz9 and xyz10 here depicts that xyz9's status has not been updated for too long)
description: replaying, master_position=[object_number=3, tag_tid=1, entry_tid=7], mirror_position=, entries_behind_master=7
last_update: 2018-10-17 15:48:09
description: replaying, master_position=[object_number=1, tag_tid=2, entry_tid=1], mirror_position=[object_number=1, tag_tid=2, entry_tid=1], entries_behind_master=0
last_update: 2018-10-17 17:13:40
Mirror status must be intact
It was observed that the mirror status of image which was forcefully promoted (while working on failover scenario) is not getting updated for indefinite time. Is this expected ?
$ sudo rbd mirror image promote data/big --force
Image promoted to primary
$ date "+%T"
$ sudo rbd mirror image status data/big
description: replaying, master_position=[object_number=257, tag_tid=4, entry_tid=131609], mirror_position=[object_number=257, tag_tid=4, entry_tid=131609], entries_behind_master=0
last_update: 2018-10-26 05:47:10
$ sudo rbd info data/big
rbd image 'big':
size 10GiB in 2560 objects
order 22 (4MiB objects)
features: layering, exclusive-lock, object-map, fast-diff, journaling
create_timestamp: Thu Oct 25 09:04:32 2018
mirroring state: enabled
mirroring global id: 9bfdf3e1-29d0-4c48-9bd1-e0ed3848d20f
mirroring primary: true
Is this expected ?
@Vasishta: do you have any additional steps to repeat this behavior? Was this cluster the primary cluster or one of the secondary clusters?
I just put down primary and one of the secondary and promoted image from remaining one secondary.
@Vasishta: "put down primary" as in shut down the cluster? That would result in the rbd-mirror daemon in-flight ops to that cluster getting hung, so at least that would explain why it wasn't getting updated.
(In reply to Jason Dillaman from comment #11)
> @Vasishta: "put down primary" as in shut down the cluster?
Yes, I had shutdown the cluster.
> That would result in the rbd-mirror daemon in-flight ops to that cluster getting hung, so at
> least that would explain why it wasn't getting updated.
Oh okay, Thanks for the clarification.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.