Bug 2101735

Summary: [rbd-mirror] : Miss in alternate snapshot schedule for some of the images which were observed through snap ls --all
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasishta <vashastr>
Component: RBD-MirrorAssignee: Ilya Dryomov <idryomov>
Status: NEW --- QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.1CC: ceph-eng-bugs, cephqe-warriors
Target Milestone: ---   
Target Release: 7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vasishta 2022-06-28 09:33:09 UTC 
Description of problem:
snapshot schedule missed thrice when observed with snap ls --all.

Version-Release number of selected component (if applicable):
16.2.7-126.el8cp

How reproducible:
Observed thrice on same image in a span of 1-1:30 hours

Steps to Reproduce:
1. Configure two clusters.
2. Configure mirroring on 150 images
3. add snapshot scheduling 

Actual results:
Miss in alternate snapshot schedule for some of the images were observed.

Expected results:
Mirror Snapshots should get created (only with possible minimal delay)

Additional info:
Observation made on some of the images
Links have cli output of machine's current time, snapshot schedule status, snap ls -
http://pastebin.test.redhat.com/1061270
http://pastebin.test.redhat.com/1061268
http://pastebin.test.redhat.com/1061267
http://pastebin.test.redhat.com/1061265
http://pastebin.test.redhat.com/1061270
Comment 6 Vasishta 2023-03-24 17:28:51 UTC 
While scrubbing BZs in 6.1 today, we decided to re-iterate through this BZ and highlight particulars that can be used to begin with to analyze.

Scrubbed available logs to cherry-pick particulars on issue that was tried to track earlier -

(In reply to Vasishta from comment #0)
> Additional info:
> Observation made on some of the images
> Links have cli output of machine's current time, snapshot schedule status,
> snap ls -
> http://pastebin.test.redhat.com/1061270
> http://pastebin.test.redhat.com/1061268
> http://pastebin.test.redhat.com/1061267
> http://pastebin.test.redhat.com/1061265
> http://pastebin.test.redhat.com/1061270

--------------------------------------------------------------------------------------
In above links 

1) http://pastebin.test.redhat.com/1061265 has `snapshot schedule status` and `snap ls --all` of image create_delete_magna006_143

In 3rd line (o/p of snapshot schedule status) it can be observed that there should have been a primary snapshot created at 04:27:00 (with acceptable delay)
But the actual next snapshot after  04:24:51 was 04:30:55



2) http://pastebin.test.redhat.com/1061267 has `snapshot schedule status` and `snap ls --all` of image create_delete_magna006_1

In 3rd (o/p of snapshot schedule status) it can be observed that there should have been a primary snapshot created at 05:57:00
But the next snap was created at 06:00:57 2022 (Last line)

3) http://pastebin.test.redhat.com/1061268

Expected - 05:33:00 (pastebin snippet doesn't have this recorded but following the pattern it can be observed)
Actual - 05:37:02 (expected is 5:36:00)

----------------------------------------------------------------------------------------