Bug 2076268

Summary: [RBD-Mirror journaling] Some of the images are "up+unknown" state from secondary cluster which are deleted from primary
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Gopi <gpatta>
Component: RBD-MirrorAssignee: Ilya Dryomov <idryomov>
Status: NEW --- QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 5.1CC: ceph-eng-bugs, idryomov, jdurgin, mmurthy
Target Milestone: ---   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 2 Gopi 2022-04-18 15:17:31 UTC 
When I try to create new image or remove an existing image from primary I am getting error:

[ceph: root@magna021 ~]# rbd create rbd_mirror_sample --size 1024 --pool rbd_mirror_pool --image-feature exclusive-lock,journaling
2022-04-18T15:11:16.918+0000 7f872a97e200 10 librbd: create name=rbd_mirror_sample, id= 27a049083765c, size=1073741824, opts=[format=2, features=68]
2022-04-18T15:11:16.925+0000 7f872a97e200 15 librbd::image::GetMetadataRequest: 0x55f89a475f70 metadata_list: start_key=conf_
2022-04-18T15:11:16.927+0000 7f870b7fe700 15 librbd::image::GetMetadataRequest: 0x55f89a475f70 handle_metadata_list: r=0
2022-04-18T15:11:16.927+0000 7f870b7fe700 15 librbd::image::GetMetadataRequest: 0x55f89a475f70 finish: r=0
2022-04-18T15:11:16.927+0000 7f872a97e200 10 librbd::image::CreateRequest: 0x55f89a2c3860 CreateRequest: name=rbd_mirror_sample, id=27a049083765c, size=1073741824, features=68, order=22, stripe_unit=0, stripe_count=0, journal_order=24, journal_splay_width=4, journal_pool=, data_pool=
2022-04-18T15:11:16.927+0000 7f872a97e200 15 librbd::image::CreateRequest: 0x55f89a2c3860 validate_data_pool: 
2022-04-18T15:11:16.927+0000 7f872a97e200  5 librbd::image::ValidatePoolRequest: read_rbd_info: 
2022-04-18T15:11:16.927+0000 7f870bfff700  5 librbd::image::ValidatePoolRequest: handle_read_rbd_info: r=0
2022-04-18T15:11:16.927+0000 7f870bfff700  5 librbd::image::ValidatePoolRequest: finish: r=0
2022-04-18T15:11:16.927+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_validate_data_pool: r=0
2022-04-18T15:11:16.927+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 add_image_to_directory: 
2022-04-18T15:11:16.931+0000 7f870b7fe700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_add_image_to_directory: r=0
2022-04-18T15:11:16.931+0000 7f870b7fe700 15 librbd::image::CreateRequest: 0x55f89a2c3860 create_id_object: 
2022-04-18T15:11:16.933+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_create_id_object: r=0
2022-04-18T15:11:16.933+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 create_image: 
2022-04-18T15:11:16.937+0000 7f870b7fe700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_create_image: r=0
2022-04-18T15:11:16.937+0000 7f870b7fe700 15 librbd::image::CreateRequest: 0x55f89a2c3860 fetch_mirror_mode: 
2022-04-18T15:11:16.938+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_fetch_mirror_mode: r=0
2022-04-18T15:11:16.938+0000 7f870bfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 journal_create: 
2022-04-18T15:11:16.947+0000 7f86fbfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_journal_create: r=0
2022-04-18T15:11:16.947+0000 7f86fbfff700 15 librbd::image::CreateRequest: 0x55f89a2c3860 mirror_image_enable: 
2022-04-18T15:11:16.947+0000 7f86fbfff700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 get_mirror_image: 
2022-04-18T15:11:16.948+0000 7f870bfff700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 handle_get_mirror_image: r=-2
2022-04-18T15:11:16.948+0000 7f870bfff700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 get_tag_owner: 
2022-04-18T15:11:16.949+0000 7f870bfff700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 handle_get_tag_owner: r=0
2022-04-18T15:11:16.949+0000 7f870bfff700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 image_state_update: 
2022-04-18T15:11:16.949+0000 7f870bfff700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 set_mirror_image: 
2022-04-18T15:11:16.952+0000 7f870b7fe700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 handle_set_mirror_image: r=0
2022-04-18T15:11:16.952+0000 7f870b7fe700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 notify_mirroring_watcher: 
2022-04-18T15:11:21.952+0000 7f870b7fe700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 handle_notify_mirroring_watcher: r=-110
2022-04-18T15:11:21.952+0000 7f870b7fe700 -1 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 handle_notify_mirroring_watcher: failed to notify mirror image update: (110) Connection timed out
2022-04-18T15:11:21.952+0000 7f870b7fe700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f86fc00dd90 finish: r=0
2022-04-18T15:11:21.952+0000 7f870b7fe700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 handle_image_state_update: r=0
2022-04-18T15:11:21.952+0000 7f870b7fe700 10 librbd::mirror::EnableRequest: 0x7f86f4002150 finish: r=0
2022-04-18T15:11:21.952+0000 7f870b7fe700 15 librbd::image::CreateRequest: 0x55f89a2c3860 handle_mirror_image_enable: r=0
2022-04-18T15:11:21.952+0000 7f870b7fe700 10 librbd::image::CreateRequest: 0x55f89a2c3860 complete: r=0
[ceph: root@magna021 ~]#


[ceph: root@magna021 ~]# rbd rm rbd_mirror_pool/rbd_mirror_image199
2022-04-18T15:08:03.808+0000 7f248b5b1200 15 librbd::image::GetMetadataRequest: 0x563c62c85350 metadata_list: start_key=conf_
2022-04-18T15:08:03.810+0000 7f24793fd700 15 librbd::image::GetMetadataRequest: 0x563c62c85350 handle_metadata_list: r=0
2022-04-18T15:08:03.810+0000 7f24793fd700 15 librbd::image::GetMetadataRequest: 0x563c62c85350 finish: r=0
2022-04-18T15:08:03.813+0000 7f248b5b1200 10 librbd::ImageCtx: 0x563c62c8bdd0 ImageCtx: image_name=, image_id=19e02e3be7f8d
2022-04-18T15:08:03.813+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62ccdc30 register_dispatch: dispatch_layer=10
2022-04-18T15:08:03.813+0000 7f248b5b1200  5 librbd::io::QueueImageDispatch: 0x563c62ccdcf0 QueueImageDispatch: ictx=0x563c62c8bdd0
2022-04-18T15:08:03.813+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62ccdc30 register_dispatch: dispatch_layer=1
2022-04-18T15:08:03.813+0000 7f248b5b1200  5 librbd::io::QosImageDispatch: 0x563c62ccdd90 QosImageDispatch: ictx=0x563c62c8bdd0
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62ccdc30 register_dispatch: dispatch_layer=2
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::RefreshImageDispatch: 0x563c62ccdab0 RefreshImageDispatch: ictx=0x563c62c8bdd0
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62ccdc30 register_dispatch: dispatch_layer=4
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::WriteBlockImageDispatch: 0x563c62ab2e30 WriteBlockImageDispatch: ictx=0x563c62c8bdd0
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62ccdc30 register_dispatch: dispatch_layer=7
2022-04-18T15:08:03.814+0000 7f248b5b1200  5 librbd::io::Dispatcher: 0x563c62c829e0 register_dispatch: dispatch_layer=6
2022-04-18T15:08:03.814+0000 7f248b5b1200 10 librbd::ImageState: 0x563c62c82960 0x563c62c82960 send_open_unlock
.
.
.
2022-04-18T15:08:03.859+0000 7f24793fd700 10 librbd::ManagedLock: 0x7f2460003008 handle_post_acquire_lock: r=0
2022-04-18T15:08:03.859+0000 7f248b5b1200 10 librbd::api::Trash: disable_mirroring: 
2022-04-18T15:08:03.859+0000 7f248b5b1200 10 librbd::mirror::DisableRequest: 0x563c62cc99d0 send_get_mirror_info: 
2022-04-18T15:08:03.859+0000 7f2479bfe700  5 librbd::Watcher: 0x7f245c006840 notifications_blocked: blocked=0
2022-04-18T15:08:03.859+0000 7f2479bfe700 10 librbd::Watcher::C_NotifyAck 0x563c62ccc370 C_NotifyAck: id=5540507815711, handle=139794139211808
2022-04-18T15:08:03.859+0000 7f2479bfe700 10 librbd::ImageWatcher: 0x7f245c006840 image exclusively locked announcement
2022-04-18T15:08:03.859+0000 7f2479bfe700 10 librbd::ImageWatcher: 0x7f245c006840 current lock owner: [145010,139794139211808]
2022-04-18T15:08:03.859+0000 7f2479bfe700 10 librbd::Watcher::C_NotifyAck 0x563c62ccc370 finish: r=0
2022-04-18T15:08:03.861+0000 7f2479bfe700 10 librbd::mirror::DisableRequest: 0x563c62cc99d0 handle_get_mirror_info: r=0
2022-04-18T15:08:03.861+0000 7f2479bfe700 10 librbd::mirror::DisableRequest: 0x563c62cc99d0 send_image_state_update: 
2022-04-18T15:08:03.861+0000 7f2479bfe700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f245c015ca0 set_mirror_image: 
2022-04-18T15:08:03.863+0000 7f24793fd700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f245c015ca0 handle_set_mirror_image: r=0
2022-04-18T15:08:03.863+0000 7f24793fd700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f245c015ca0 notify_mirroring_watcher: 
2022-04-18T15:08:08.865+0000 7f24793fd700 10 librbd::mirror::ImageStateUpdateRequest: 0x7f245c015ca0 handle_notify_mirroring_watcher: r=-110
2022-04-18T15:08:08.865+0000 7f24793fd700 -1 librbd::mirror::ImageStateUpdateRequest: 0x7f245c015ca0 handle_notify_mirroring_watcher: failed to notify mirror image update: (110) Connection timed out
.
.
.
2022-04-18T15:08:13.958+0000 7f24793fd700  5 librbd::io::Dispatcher: 0x7f246c00ac80 shut_down: 
2022-04-18T15:08:13.958+0000 7f24793fd700  5 librbd::io::ObjectDispatch: 0x7f246004ed10 shut_down: 
2022-04-18T15:08:13.958+0000 7f24793fd700  5 librbd::io::SimpleSchedulerObjectDispatch: 0x7f245c027460 shut_down: 
2022-04-18T15:08:13.958+0000 7f24793fd700  5 librbd::cache::WriteAroundObjectDispatch: 0x7f245c0265c0 shut_down: 
2022-04-18T15:08:13.958+0000 7f24793fd700 10 librbd::image::CloseRequest: 0x7f2460062380 handle_shut_down_object_dispatcher: r=0
2022-04-18T15:08:13.958+0000 7f24793fd700 10 librbd::image::CloseRequest: 0x7f2460062380 send_flush_op_work_queue
2022-04-18T15:08:13.958+0000 7f24793fd700 10 librbd::image::CloseRequest: 0x7f2460062380 handle_flush_op_work_queue: r=0
2022-04-18T15:08:13.958+0000 7f24793fd700 10 librbd::image::CloseRequest: 0x7f2460062380 handle_flush_image_watcher: r=0
2022-04-18T15:08:13.958+0000 7f24793fd700 10 librbd::ImageState: 0x7f245c00d8c0 0x7f245c00d8c0 handle_close: r=0
2022-04-18T15:08:13.958+0000 7f245bfff700 10 librbd::ImageCtx: 0x7f246000a8d0 ~ImageCtx
2022-04-18T15:08:13.974+0000 7f24793fd700 10 librbd::trash::RemoveRequest: 0x563c62cc7490 handle_remove_image: r=0
2022-04-18T15:08:13.974+0000 7f24793fd700 10 librbd::trash::RemoveRequest: 0x563c62cc7490 remove_trash_entry: 
2022-04-18T15:08:13.976+0000 7f2479bfe700 10 librbd::trash::RemoveRequest: 0x563c62cc7490 handle_remove_trash_entry: r=0
2022-04-18T15:08:13.976+0000 7f2479bfe700 10 librbd::trash::RemoveRequest: 0x563c62cc7490 finish: r=0
Removing image: 100% complete...done.


Note: I pasted only one image output as i can't paste for all images, since there are more than 200 images under pool.