Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1947224

Summary: [RBD] - Ceph-immutable-object-cache "Parent cache fail to connect RO daemon"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Harish Munjulur <hmunjulu>
Component: RBDAssignee: Ilya Dryomov <idryomov>
Status: CLOSED ERRATA QA Contact: Harish Munjulur <hmunjulu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0CC: ceph-eng-bugs, idryomov, kdreyer, tserlin
Target Milestone: ---   
Target Release: 5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-30 08:29:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ceph grep
none
output of rbd export none

Description Harish Munjulur 2021-04-08 00:53:25 UTC 
Description of problem:
[ceph: root@magna031 ~]# rbd export rbd_clone_pool0/cloned_image0 - | md5sum
2021-04-08T00:30:12.192+0000 7f6143fff700 -1 librbd::cache::ParentCacheObjectDispatch: 0x7f6154020710 operator(): Parent cache fail to connect RO daeomn.


Version-Release number of selected component (if applicable): ceph version 16.2.0-4.el8cp


How reproducible: 2/2


Steps to Reproduce:
1. Enable ceph immutable object cache daemon and enable the flags
rbd parent cache enabled = true
rbd plugins = parent_cache

2. Create a parent primary_image and pool. Write data to image
3. create a new pool and clone the primary_image 
4. check md5 on the cloned image

Actual results:

on the primary
[ceph: root@magna031 ~]# rbd export rbd_primary_pool/image_test - | md5sum
Exporting image: 100% complete...done.
9bcb92d5c4bbd198e1f532d905245a33  -

on the cloned image:
[ceph: root@magna031 ~]# rbd export rbd_clone_pool0/cloned_image0 - | md5sum
2021-04-08T00:30:12.192+0000 7f6143fff700 -1 librbd::cache::ParentCacheObjectDispatch: 0x7f6154020710 operator(): Parent cache fail to connect RO daeomn.
2021-04-08T00:30:12.192+0000 7f6143fff700 -1 librbd::cache::ParentCacheObjectDispatch: 0x7f6154020710 operator(): Parent cache fail to register client.
2021-04-08T00:30:12.192+0000 7f6143fff700 -1 librbd::cache::ParentCacheObjectDispatch: 0x7f6154020710 handle_register_client: Parent cache register fails.
Exporting image: 3% complete...2021-04-08T00:30:12.194+0000 7f6143fff700 -1 librbd::cache::ParentCacheObjectDispatch: 0x7f6154020710 operator(): Parent cache fail to connect RO daeomn.


Expected results: Should not show any error
Comment 2 Harish Munjulur 2021-04-20 02:46:02 UTC 
1. Was ceph-immutable-object-cache daemon running?

Yes, its active and Running. 

2. Was it and "rbd export" using the same ceph.conf file?

Running inside the cephadm shell. So must be using the ceph.conf file. 

3. Please paste the output of "ps aux | grep ceph" and "stat /var/run/ceph/immutable_object_cache_sock" right before "rbd export" is executed. Run "rbd export" with --debug_rbd=25 --debug_immutable_obj_cache=25.


Attached are the logs for "ps aux | grep ceph"

[root@magna031 ~]# stat /var/run/ceph/immutable_object_cache_sock
  File: /var/run/ceph/immutable_object_cache_sock
  Size: 0         	Blocks: 0          IO Block: 4096   socket
Device: 18h/24d	Inode: 1601607     Links: 1
Access: (0755/srwxr-xr-x)  Uid: (  167/    ceph)   Gid: (  167/    ceph)
Context: system_u:object_r:var_run_t:s0
Access: 2021-04-09 20:01:31.756874490 +0000
Modify: 2021-04-09 20:01:31.756874490 +0000
Change: 2021-04-09 20:01:31.756874490 +0000
 Birth: -
Comment 3 Harish Munjulur 2021-04-20 02:47:11 UTC 
Created attachment 1773606 [details]
ceph grep
Comment 4 Harish Munjulur 2021-04-20 02:48:40 UTC 
Created attachment 1773607 [details]
output of rbd export
Comment 6 Harish Munjulur 2021-04-21 10:04:37 UTC 
Sure Thanks Ilya for the right steps. I will test it outside the cephadm and close this BZ after verification.
Comment 7 Ken Dreyer (Red Hat) 2021-04-26 21:03:46 UTC 
(I've posted https://github.com/ceph/ceph/pull/41035 for the error message spelling.)
Comment 8 Harish Munjulur 2021-04-28 15:06:48 UTC 
QA Verified. Works as expected outside cephadm shell
Comment 11 errata-xmlrpc 2021-08-30 08:29:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294