Bug 2362859

Summary:	[CephFS - FScrypt] Snapshot directory .snap has junk/non-readable chars in unlocked mode after snapshot data copy op
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	sumr
Component:	CephFS	Assignee:	Christopher Hoffman <choffman>
Status:	CLOSED ERRATA	QA Contact:	sumr
Severity:	high	Docs Contact:	Rivka Pollack <rpollack>
Priority:	unspecified
Version:	8.1	CC:	ceph-eng-bugs, cephqe-warriors, choffman, hyelloji, ngangadh, pdonnell, rpollack, tserlin
Target Milestone:	---
Target Release:	8.1
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	ceph-19.2.1-187.el9cp	Doc Type:	Bug Fix
Doc Text:	Snapshot names are now stored in plain text Previously, snapshots could be created regardless of whether the `fscrypt` key was present. When a snapshot was created using the `mgr subvolume snapshot create` command without the key, the snapshot name was not encrypted during creation. As a result, subsequent attempts to decrypt the plain text name produced unreadable output. With this fix, snapshot names are stored as plain text without encryption. This change helps ensure that snapshot names remain readable, whether the `fscrypt` key is present or not.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2025-06-26 12:31:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2351689

Description sumr 2025-04-29 08:40:11 UTC

Description of problem:
Snapshot directory .snap contents are non-readable and junk characters when listed from encrypt enabled directory in unlocked mode.

[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# fscrypt status ../testdir2
"../testdir2" is encrypted with fscrypt.

Policy:   0afab0d80eb66f78d5b9dea66540622c
Options:  padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
Unlocked: Yes

Protected with 1 protector:
PROTECTOR         LINKED  DESCRIPTION
d87493865735878a  No      raw key protector "cephfs"
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -l
total 2404871
-rw-r--r--. 1 root root     948290 Apr 29 07:24 Chorus.wav
-rw-r--r--. 1 root root    2372820 Apr 29 07:24 ForBiggerJoyrides.mp4
drwxr-xr-x. 3 root root   12115968 Apr 29 07:24 file_dstdir
drwxr-xr-x. 3 root root   12288000 Apr 29 07:24 file_srcdir
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 fio_file1.0.0
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 fio_file1.2.0
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 fio_file1.3.0
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 fio_file1.4.0
-rw-r--r--. 1 root root     260733 Apr 29 07:25 image-1.jpg
drwxr-xr-x. 2 root root       8192 Apr 29 07:25 network_shared
-rw-r--r--. 1 root root      65711 Apr 29 07:25 plain_text_1m
-rw-r--r--. 1 root root          2 Jan  1  1970 smallfile
-rw-r--r--. 1 root root    5242880 Apr 29 07:25 sparse_file_5m
-rw-r--r--. 1 root root 2411750806 Apr 29 07:25 symlink_plain_text_2g
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -al .snap
ls: cannot access '.snap/'$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺': No such file or directory
total 1
d?????????? ? ?    ?             ?            ? ''$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺'
drwx------. 2 root root          0 Apr 29 07:27  .
drwx------. 5 root root 2487013376 Apr 29 07:26  ..
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -al .snap/^^6?O?\"u????????^ST턺 
ls: cannot access '.snap/'$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺': No such file or directory

Version-Release number of selected component (if applicable): 19.2.1-161.el9cp 


How reproducible:


Steps to Reproduce:
1. Create subvolume and perform ceph-fuse mount.
2. Create testdir and enable fscrypt encrypt, add data, and lock the directory
3. Create snapshot and view the contents of snapshot in locked mode across different paths.
Observation: testdir contents read as encrypted within snapshot directory under testdir
4. Try copying snapshot contents in locked mode and validate read error.
4. Unlock testdir, Copy the testdir from snapshot created to another subvolume which is not encrypt enabled
5. Try reading the .snap contents from parent subvolume's testdir again.

Actual results: .snap contents from parent testdir in unlocked mode is printed as junk and non-readable


Expected results: In unlocked mode, .snap contents should be readable and appropriate snapshot name should exist.


Additional info:

Logs:

[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# fscrypt lock ../testdir2
"../testdir2" is now locked.
[root@ceph-sumar-fscrypt-az0v8f-node6 ~]# ceph fs subvolume snapshot create cephfs sv5 snap1_locked
[root@ceph-sumar-fscrypt-az0v8f-node6 ~]# 
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# cd .snap
[root@ceph-sumar-fscrypt-az0v8f-node6 .snap]# ls
_snap1_locked_1099511637823
[root@ceph-sumar-fscrypt-az0v8f-node6 .snap]# cd _snap1_locked_1099511637823/
[root@ceph-sumar-fscrypt-az0v8f-node6 _snap1_locked_1099511637823]# ls -l
total 2404871
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 08aEgY37T2i,zth2d,g,POuPOkn28l,jHI+aqNjMA5U
-rw-r--r--. 1 root root   10485760 Apr 29 07:25 AhMN99SBuoy084Ag,FI2cxmx9Tqbrbub1LjgIDVk6gM

[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# cp -rf .snap/_snap1_locked_1099511637823/testdir2/ /mnt/fuse_sv6/
cp: overwrite '/mnt/fuse_sv6/testdir2/Chorus.wav'? n
cp: overwrite '/mnt/fuse_sv6/testdir2/ForBiggerJoyrides.mp4'? n
^C
[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]#
[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# cp -rf .snap/_snap1_locked_1099511637823/testdir2/fio* /mnt/fuse_sv6/testdir2/
[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# cp -rf .snap/_snap1_locked_1099511637823/testdir2/smallfile /mnt/fuse_sv6/testdir2/
[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# cp -rf .snap/_snap1_locked_1099511637823/testdir2/sparse_file_5m /mnt/fuse_sv6/testdir2/
[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# cp -rf .snap/_snap1_locked_1099511637823/testdir2/image-1.jpg /mnt/fuse_sv6/testdir2/

[root@ceph-sumar-fscrypt-az0v8f-node6 ~]# cp -rf /mnt/fuse_sv5/.snap/_snap1_locked_1099511637823/testdir2/ /mnt/fuse_sv5/testdir3/

[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# fscrypt status ../testdir2
"../testdir2" is encrypted with fscrypt.

Policy:   0afab0d80eb66f78d5b9dea66540622c
Options:  padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
Unlocked: Yes

Protected with 1 protector:
PROTECTOR         LINKED  DESCRIPTION
d87493865735878a  No      raw key protector "cephfs"
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -l
total 2404871
-rw-r--r--. 1 root root     948290 Apr 29 07:24 Chorus.wav
-rw-r--r--. 1 root root    2372820 Apr 29 07:24 ForBiggerJoyrides.mp4

[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -al .snap
ls: cannot access '.snap/'$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺': No such file or directory
total 1
d?????????? ? ?    ?             ?            ? ''$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺'
drwx------. 2 root root          0 Apr 29 07:27  .
drwx------. 5 root root 2487013376 Apr 29 07:26  ..
[root@ceph-sumar-fscrypt-az0v8f-node6 testdir2]# ls -al .snap/^^6?O?\"u????????^ST턺 
ls: cannot access '.snap/'$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺': No such file or directory

Snapshot contents are readable from non-encrypt directory,

[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# ls /mnt/fuse_sv5/.snap
_snap1_locked_1099511637823

Not from encrypt directory in unlocked mode,

[root@ceph-sumar-fscrypt-az0v8f-node6 fuse_sv5]# ls /mnt/fuse_sv5/testdir2/.snap
ls: cannot access '/mnt/fuse_sv5/testdir2/.snap/'$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺': No such file or directory
''$'\036''6'$'\262''O'$'\375''"u'$'\271\264\221\367\370\243\276\206\023''T턺'

Comment 14 errata-xmlrpc 2025-06-26 12:31:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775