Bug 2343918

Summary: cephfs_mirror: incremental sync results in unidentical snapshots
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Jos Collin <jcollin>
Component: CephFSAssignee: Venky Shankar <vshankar>
Status: CLOSED ERRATA QA Contact: Hemanth Kumar <hyelloji>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.1CC: ceph-eng-bugs, cephqe-warriors, gfarnum, ngangadh, tserlin
Target Milestone: ---   
Target Release: 8.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ceph-19.2.1-73.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-06-26 12:24:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jos Collin 2025-02-05 11:59:02 UTC
Description of problem:
Incremental sync results in nonidentical source and destination snapshots for:
1. Incremental Syncing of already existing snapshots.
2. test_cephfs_mirror_incremental_sync in qa/tasks/cephfs/test_mirroring.py.

Version-Release number of selected component (if applicable):
8.1

How reproducible:
Intermittently. 
These issues are not always reproducible.

Steps to Reproduce:
1.
Incremental Syncing of already existing snapshots
-------------------------------------

Mount:
```
$ sudo ./bin/mount.ceph admin@.a=/ /mnt/cephfs1 -o conf=$(pwd)/ceph.conf;
$ sudo ./bin/mount.ceph admin@.remotefs=/ /mnt/cephfs2 -o conf=$(pwd)/ceph.conf;
```

Make some changes and take a snapshot:
```
$ mkdir -p /mnt/cephfs1/dir/d1
$ touch /mnt/cephfs1/dir/d1/file1
$ mkdir -p /mnt/cephfs1/dir/.snap/snap1; 
```

Make some more changes & take another snapshot:
```
$ touch /mnt/cephfs1/dir/d1/file2
$ mkdir -p /mnt/cephfs1/dir/.snap/snap2;
```

Make a change and don't take snapshot:
`
$ rm /mnt/cephfs1/dir/d1/file2
`

Add `dir` for mirroring:
`
$ ./bin/ceph fs snapshot mirror add a /dir
`

Results:
```
$ tree /mnt/cephfs1/dir/.snap
/mnt/cephfs1/dir/.snap
├── snap1
│   └── d1
│       └── file1
└── snap2
    └── d1
        ├── file1
        └── file2
```

```
$ tree /mnt/cephfs2/dir/.snap
/mnt/cephfs2/dir/.snap
├── snap1
│   └── d1
│       └── file1
└── snap2
    └── d1
        └── file1
```

```
$ tree /mnt/cephfs1/dir
/mnt/cephfs1/dir
└── d1
    └── file1
```

```
$ tree /mnt/cephfs2/dir
/mnt/cephfs2/dir
└── d1
    └── file1
```

2.
test_cephfs_mirror_incremental_sync in qa/tasks/cephfs/test_mirroring.py:
------------------------------------------------------------------------
The steps of test_cephfs_mirror_incremental_sync produces nonidentical source and destination snapshots after incremental sync of a repo (with .git directory). The checksum of .git directory differs.

Actual results:
1.
In the above example, /mnt/cephfs1/dir/.snap/snap2 and /mnt/cephfs2/dir/.snap/snap2 are different which is a bug.

2.
Incremental sync of a repo produces nonidentical source and destination snapshots.

Expected results:
1.
/mnt/cephfs1/dir/.snap/snap2 and /mnt/cephfs2/dir/.snap/snap2 should be identical.

2.
Incremental sync of a repo should produce identical source and destination snapshots.

Additional info:

Comment 4 Greg Farnum 2025-05-05 16:10:01 UTC
Moving to ON_QA as this was fixed incidentally in https://bugzilla.redhat.com/show_bug.cgi?id=231773

Feel free to adjust status as desired with that information. :)

Comment 5 Greg Farnum 2025-05-05 16:18:01 UTC
Whoops, I meant https://bugzilla.redhat.com/show_bug.cgi?id=2317735

Comment 11 errata-xmlrpc 2025-06-26 12:24:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775