Bug 1388541

Summary: [rbd-mirror] : snapshot name is different than given name( '.rbd-mirror.0f454219-dd15-49ba-968a-f822833d46e8.87ea3ebe-1c4a-4bdb-8942-245196739d51' instead of 'snap<number>')
Product: Red Hat Ceph Storage Reporter: Rachana Patel <racpatel>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Parikshith <pbyregow>
Severity: low Docs Contact:
Priority: medium    
Version: 2.1CC: anharris, ceph-eng-bugs, hnallurv
Target Milestone: rc   
Target Release: 3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-12.1.2-1.el7cp Ubuntu: ceph_12.1.2-2redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-05 23:31:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rachana Patel 2016-10-25 15:18:38 UTC 
Description of problem:
=======================
was verifying bug 1347664 and as part of it created and deleted snapshot of image on master/local cluster.(enable/disable journaling feature)

On listing Snap name was '.rbd-mirror.6dfceee6-6dd3-40dc-8611-ba65a66eee86.c64d71b6-99b4-4c45-9031-f90c429916a8' instead of "snap<number>"


Test 1:-

Version-Release number of selected component (if applicable):
===============================================================
10.2.3-8.el7cp.x86_64

How reproducible:
=================
1/2


Steps to Reproduce:
==================
1. Create an Image with Journal Enabled.
2. Create 25 snapshots.
3. While snapshot was getting synced, disable Journal.
4. After a while again enable Journal, the remaining snapshot gets synced.
5. Then delete 15 snapshots, while deletion was happening disable Journal.
6. After a while again enable Journal.
7. create snapshot with same name as deleted snapshot. 
8. disable journal and enable jornal again while snap is getting synced.
9. create snapshot with same name as deleted snapshot. 

{
for i in {1..25}; do rbd snap create pool1/bug2@snap$i --cluster master; done
rbd feature disable pool1/bug2 journaling --cluster master
rbd feature enable pool1/bug2 journaling --cluster master
for i in {1..15}; do rbd snap rm pool1/bug2@snap$i --cluster master; done
rbd feature disable pool1/bug2 journaling --cluster master
rbd feature enable pool1/bug2 journaling --cluster master
for i in {1..25}; do rbd snap create pool1/bug2@snap$i --cluster master; done
rbd feature disable pool1/bug2 journaling --cluster master
rbd feature enable pool1/bug2 journaling --cluster master
for i in {1..15}; do rbd snap rm pool1/bug2@snap$i --cluster master; done
rbd feature enable pool1/bug2 journaling --cluster master
}

Actual results:
===============
[root@magna025 ubuntu]#  rbd snap ls pool1/bug2 --cluster master
SNAPID NAME                                             SIZE 
   669 snap16                                       10240 MB 
   670 snap17                                       10240 MB 
   671 snap18                                       10240 MB 
   672 snap19                                       10240 MB 
   673 snap20                                       10240 MB 
   674 snap21                                       10240 MB 
   675 snap22                                       10240 MB 
   676 snap23                                       10240 MB 
   677 snap24                                       10240 MB 
   678 snap25                                       10240 MB 
   719 .rbd-mirror.0f454219-dd15-49ba-968a-f822833d46e8.87ea3ebe-1c4a-4bdb-8942-245196739d51 10240 MB 




on slave:-
[root@magna078 ubuntu]# rbd snap ls pool1/bug2 --cluster slave
SNAPID NAME                                             SIZE 
   649 snap16                                       10240 MB 
   650 snap17                                       10240 MB 
   651 snap18                                       10240 MB 
   652 snap19                                       10240 MB 
   653 snap20                                       10240 MB 
   654 snap21                                       10240 MB 
   655 snap22                                       10240 MB 
   656 snap23                                       10240 MB 
   657 snap24                                       10240 MB 
   658 snap25                                       10240 MB 
   663 .rbd-mirror.0f454219-dd15-49ba-968a-f822833d46e8.87ea3ebe-1c4a-4bdb-8942-245196739d51 10240 MB 

[root@magna078 ubuntu]# systemctl status -l ceph-rbd-mirror@slave
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-10-21 18:44:27 UTC; 2h 30min ago
 Main PID: 11803 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─11803 /usr/bin/rbd-mirror -f --cluster slave --id slave --setuser ceph --setgroup ceph

Oct 21 21:12:58 magna078 rbd-mirror[11803]: 2016-10-21 21:12:58.100415 7fce6e7fc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7fceac026e90 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Oct 21 21:12:58 magna078 rbd-mirror[11803]: 2016-10-21 21:12:58.100437 7fcebcf90700 -1 rbd::mirror::ImageReplayer: 0x7fce3805ef80 [3/07737341-c9b6-45ec-9f05-755864b36d68] operator(): start failed: (2) No such file or directory
Oct 21 21:13:49 magna078 rbd-mirror[11803]: 2016-10-21 21:13:49.175451 7fce6e7fc700 -1 rbd::mirror::image_sync::SnapshotCopyRequest: 0x7fce000ad240 handle_update_client: failed to update client data: (2) No such file or directory
Oct 21 21:13:49 magna078 rbd-mirror[11803]: 2016-10-21 21:13:49.175466 7fce6e7fc700 -1 rbd::mirror::ImageSync: 0x7fce1c0d0590 handle_copy_snapshots: failed to copy snapshot metadata: (2) No such file or directory
Oct 21 21:13:49 magna078 rbd-mirror[11803]: 2016-10-21 21:13:49.175473 7fce6e7fc700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7fceac026e90 handle_image_sync: failed to sync remote image: (2) No such file or directory
Oct 21 21:13:49 magna078 rbd-mirror[11803]: 2016-10-21 21:13:49.194938 7fcebcf90700 -1 rbd::mirror::ImageReplayer: 0x7fce3805ef80 [3/e709ade3-79c1-40c0-b260-0dcff79352a2] operator(): start failed: (2) No such file or directory
Oct 21 21:13:59 magna078 rbd-mirror[11803]: 2016-10-21 21:13:59.686362 7fce6cff9700 -1 rbd::mirror::image_sync::SnapshotCopyRequest: 0x7fce000b5dd0 handle_snap_remove: failed to remove snapshot 'snap2': (16) Device or resource busy
Oct 21 21:13:59 magna078 rbd-mirror[11803]: 2016-10-21 21:13:59.686372 7fce6cff9700 -1 rbd::mirror::ImageSync: 0x7fcdd805c310 handle_copy_snapshots: failed to copy snapshot metadata: (16) Device or resource busy
Oct 21 21:13:59 magna078 rbd-mirror[11803]: 2016-10-21 21:13:59.686380 7fce6cff9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7fceac026e90 handle_image_sync: failed to sync remote image: (16) Device or resource busy
Oct 21 21:13:59 magna078 rbd-mirror[11803]: 2016-10-21 21:13:59.707309 7fcebcf90700 -1 rbd::mirror::ImageReplayer: 0x7fce3805ef80 [3/e709ade3-79c1-40c0-b260-0dcff79352a2] operator(): start failed: (16) Device or resource busy


Note:-
Snap remained on both cluster till i removed it from both cluster

Expected results:
=================
on both cluster snap name should be snap<number> not  .rbd-mirror.0f454219-dd15-49ba-968a-f822833d46e8.87ea3ebe-1c4a-4bdb-8942-245196739d51




Test 2:-
========
Version-Release number of selected component (if applicable):
===============================================================
10.2.3-10.el7cp.x86_64

How reproducible:
=================
intermittent


Steps to Reproduce:
==================
same as above



Actual results:
===============

Issue was found on slave/remote cluster only. Master/Local cluster didnot have any snap having weird name

[root@magna078 ubuntu]#  rbd snap ls pool1/bug2 --cluster slave
2016-10-25 13:51:21.027431 7f6481f3ed80  5 librbd::AioImageRequestWQ: 0x7f648bff2830 : ictx=0x7f648bff0770
2016-10-25 13:51:21.027505 7f6481f3ed80 20 librbd::ImageState: 0x7f648bff1a00 open
2016-10-25 13:51:21.027512 7f6481f3ed80 10 librbd::ImageState: 0x7f648bff1a00 0x7f648bff1a00 send_open_unlock
2016-10-25 13:51:21.027520 7f6481f3ed80 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_v2_detect_header
2016-10-25 13:51:21.029005 7f645e7fc700 10 librbd::image::OpenRequest: handle_v2_detect_header: r=0
2016-10-25 13:51:21.029013 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_v2_get_id
2016-10-25 13:51:21.029571 7f645e7fc700 10 librbd::image::OpenRequest: handle_v2_get_id: r=0
2016-10-25 13:51:21.029579 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_v2_get_immutable_metadata
2016-10-25 13:51:21.031029 7f645e7fc700 10 librbd::image::OpenRequest: handle_v2_get_immutable_metadata: r=0
2016-10-25 13:51:21.031040 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_v2_get_stripe_unit_count
2016-10-25 13:51:21.031562 7f645e7fc700 10 librbd::image::OpenRequest: handle_v2_get_stripe_unit_count: r=-8
2016-10-25 13:51:21.031578 7f645e7fc700 10 librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.10733855585c format rbd_data.10733855585c.%016llx
2016-10-25 13:51:21.031584 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_v2_apply_metadata: start_key=conf_
2016-10-25 13:51:21.032108 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 handle_v2_apply_metadata: r=0
2016-10-25 13:51:21.032117 7f645e7fc700 20 librbd::ImageCtx: apply_metadata
2016-10-25 13:51:21.032257 7f645e7fc700 20 librbd::ImageCtx: enabling caching...
2016-10-25 13:51:21.032261 7f645e7fc700 20 librbd::ImageCtx: Initial cache settings: size=33554432 num_objects=10 max_dirty=25165824 target_dirty=16777216 max_dirty_age=1
2016-10-25 13:51:21.032325 7f645e7fc700 10 librbd::ImageCtx:  cache bytes 33554432 -> about 1048 objects
2016-10-25 13:51:21.032357 7f645e7fc700 10 librbd::image::OpenRequest: 0x7f648bff31a0 send_refresh
2016-10-25 13:51:21.032364 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 send_v2_get_mutable_metadata
2016-10-25 13:51:21.033161 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 handle_v2_get_mutable_metadata: r=0
2016-10-25 13:51:21.033181 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 send_v2_get_flags
2016-10-25 13:51:21.034264 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 handle_v2_get_flags: r=0
2016-10-25 13:51:21.034277 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 send_v2_get_snapshots
2016-10-25 13:51:21.037544 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 handle_v2_get_snapshots: r=0
2016-10-25 13:51:21.037568 7f645e7fc700 10 librbd::image::RefreshRequest: 0x7f6444003c30 send_v2_apply
2016-10-25 13:51:21.037583 7f645dffb700 10 librbd::image::RefreshRequest: 0x7f6444003c30 handle_v2_apply
2016-10-25 13:51:21.037594 7f645dffb700 20 librbd::image::RefreshRequest: 0x7f6444003c30 apply
2016-10-25 13:51:21.037599 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=83 name=snap1 size=10737418240
2016-10-25 13:51:21.037603 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=82 name=.rbd-mirror.6dfceee6-6dd3-40dc-8611-ba65a66eee86.c64d71b6-99b4-4c45-9031-f90c429916a8 size=10737418240
2016-10-25 13:51:21.037605 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=81 name=snap25 size=10737418240
2016-10-25 13:51:21.037606 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=80 name=snap24 size=10737418240
2016-10-25 13:51:21.037607 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=79 name=snap23 size=10737418240
2016-10-25 13:51:21.037608 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=78 name=snap22 size=10737418240
2016-10-25 13:51:21.037609 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=77 name=snap21 size=10737418240
2016-10-25 13:51:21.037611 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=76 name=snap20 size=10737418240
2016-10-25 13:51:21.037611 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=75 name=snap19 size=10737418240
2016-10-25 13:51:21.037612 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=74 name=snap18 size=10737418240
2016-10-25 13:51:21.037613 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=73 name=snap17 size=10737418240
2016-10-25 13:51:21.037614 7f645dffb700 20 librbd::image::RefreshRequest: new snapshot id=72 name=snap16 size=10737418240
2016-10-25 13:51:21.037634 7f645dffb700 10 librbd::image::RefreshRequest: 0x7f6444003c30 send_flush_aio
2016-10-25 13:51:21.037646 7f645dffb700 10 librbd::image::RefreshRequest: 0x7f6444003c30 handle_flush_aio: r=0
2016-10-25 13:51:21.037652 7f645dffb700 10 librbd::image::OpenRequest: handle_refresh: r=0
2016-10-25 13:51:21.037655 7f645dffb700 10 librbd::ImageState: 0x7f648bff1a00 0x7f648bff1a00 handle_open: r=0
2016-10-25 13:51:21.037686 7f6481f3ed80 20 librbd: snap_list 0x7f648bff0770
SNAPID NAME                                                                                      SIZE 
    72 snap16                                                                                10240 MB 
    73 snap17                                                                                10240 MB 
    74 snap18                                                                                10240 MB 
    75 snap19                                                                                10240 MB 
    76 snap20                                                                                10240 MB 
    77 snap21                                                                                10240 MB 
    78 snap22                                                                                10240 MB 
    79 snap23                                                                                10240 MB 
    80 snap24                                                                                10240 MB 
    81 snap25                                                                                10240 MB 
    82 .rbd-mirror.6dfceee6-6dd3-40dc-8611-ba65a66eee86.c64d71b6-99b4-4c45-9031-f90c429916a8 10240 MB 
    83 snap1                                                                                 10240 MB 
2016-10-25 13:51:21.037806 7f6481f3ed80 20 librbd::ImageState: 0x7f648bff1a00 close
2016-10-25 13:51:21.037811 7f6481f3ed80 10 librbd::ImageState: 0x7f648bff1a00 0x7f648bff1a00 send_close_unlock
2016-10-25 13:51:21.037816 7f6481f3ed80 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_shut_down_update_watchers
2016-10-25 13:51:21.037818 7f6481f3ed80 20 librbd::ImageState: 0x7f648bff1a00 shut_down_update_watchers
2016-10-25 13:51:21.037819 7f6481f3ed80 20 librbd::ImageState: 0x7f648bff1b00 ImageUpdateWatchers::shut_down
2016-10-25 13:51:21.037822 7f6481f3ed80 20 librbd::ImageState: 0x7f648bff1b00 ImageUpdateWatchers::shut_down: completing shut down
2016-10-25 13:51:21.037833 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_shut_down_update_watchers: r=0
2016-10-25 13:51:21.037836 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_shut_down_aio_queue
2016-10-25 13:51:21.037838 7f645dffb700  5 librbd::AioImageRequestWQ: shut_down: in_flight=0
2016-10-25 13:51:21.037842 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_shut_down_aio_queue: r=0
2016-10-25 13:51:21.037846 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_flush
2016-10-25 13:51:21.037849 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_flush: r=0
2016-10-25 13:51:21.037850 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_flush_readahead
2016-10-25 13:51:21.037853 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_flush_readahead: r=0
2016-10-25 13:51:21.037854 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_shut_down_cache
2016-10-25 13:51:21.037897 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_shut_down_cache: r=0
2016-10-25 13:51:21.037900 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 send_flush_op_work_queue
2016-10-25 13:51:21.037902 7f645dffb700 10 librbd::image::CloseRequest: 0x7f648bff2e50 handle_flush_op_work_queue: r=0
2016-10-25 13:51:21.037912 7f645dffb700 10 librbd::ImageState: 0x7f648bff1a00 0x7f648bff1a00 handle_close: r=0
[root@magna078 ubuntu]#  rbd snap ls pool1/bug2 --cluster slave
2016-10-25 13:51:28.614734 7fe2ac2ded80  5 librbd::AioImageRequestWQ: 0x7fe2b73fa870 : ictx=0x7fe2b73f87b0
2016-10-25 13:51:28.614825 7fe2ac2ded80 20 librbd::ImageState: 0x7fe2b73f9a40 open
2016-10-25 13:51:28.614832 7fe2ac2ded80 10 librbd::ImageState: 0x7fe2b73f9a40 0x7fe2b73f9a40 send_open_unlock
2016-10-25 13:51:28.614840 7fe2ac2ded80 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_v2_detect_header
2016-10-25 13:51:28.616159 7fe27bfff700 10 librbd::image::OpenRequest: handle_v2_detect_header: r=0
2016-10-25 13:51:28.616166 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_v2_get_id
2016-10-25 13:51:28.616640 7fe27bfff700 10 librbd::image::OpenRequest: handle_v2_get_id: r=0
2016-10-25 13:51:28.616651 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_v2_get_immutable_metadata
2016-10-25 13:51:28.618010 7fe27bfff700 10 librbd::image::OpenRequest: handle_v2_get_immutable_metadata: r=0
2016-10-25 13:51:28.618019 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_v2_get_stripe_unit_count
2016-10-25 13:51:28.618559 7fe27bfff700 10 librbd::image::OpenRequest: handle_v2_get_stripe_unit_count: r=-8
2016-10-25 13:51:28.618577 7fe27bfff700 10 librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.10733855585c format rbd_data.10733855585c.%016llx
2016-10-25 13:51:28.618581 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_v2_apply_metadata: start_key=conf_
2016-10-25 13:51:28.619059 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 handle_v2_apply_metadata: r=0
2016-10-25 13:51:28.619077 7fe27bfff700 20 librbd::ImageCtx: apply_metadata
2016-10-25 13:51:28.619239 7fe27bfff700 20 librbd::ImageCtx: enabling caching...
2016-10-25 13:51:28.619246 7fe27bfff700 20 librbd::ImageCtx: Initial cache settings: size=33554432 num_objects=10 max_dirty=25165824 target_dirty=16777216 max_dirty_age=1
2016-10-25 13:51:28.619329 7fe27bfff700 10 librbd::ImageCtx:  cache bytes 33554432 -> about 1048 objects
2016-10-25 13:51:28.619379 7fe27bfff700 10 librbd::image::OpenRequest: 0x7fe2b73fb1e0 send_refresh
2016-10-25 13:51:28.619391 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 send_v2_get_mutable_metadata
2016-10-25 13:51:28.620260 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 handle_v2_get_mutable_metadata: r=0
2016-10-25 13:51:28.620281 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 send_v2_get_flags
2016-10-25 13:51:28.621673 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 handle_v2_get_flags: r=0
2016-10-25 13:51:28.621683 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 send_v2_get_snapshots
2016-10-25 13:51:28.626305 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 handle_v2_get_snapshots: r=0
2016-10-25 13:51:28.626330 7fe27bfff700 10 librbd::image::RefreshRequest: 0x7fe264003c30 send_v2_apply
2016-10-25 13:51:28.626347 7fe27b7fe700 10 librbd::image::RefreshRequest: 0x7fe264003c30 handle_v2_apply
2016-10-25 13:51:28.626360 7fe27b7fe700 20 librbd::image::RefreshRequest: 0x7fe264003c30 apply
2016-10-25 13:51:28.626365 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=89 name=snap7 size=10737418240
2016-10-25 13:51:28.626372 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=88 name=snap6 size=10737418240
2016-10-25 13:51:28.626373 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=87 name=snap5 size=10737418240
2016-10-25 13:51:28.626374 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=86 name=snap4 size=10737418240
2016-10-25 13:51:28.626375 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=85 name=snap3 size=10737418240
2016-10-25 13:51:28.626375 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=84 name=snap2 size=10737418240
2016-10-25 13:51:28.626376 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=83 name=snap1 size=10737418240
2016-10-25 13:51:28.626377 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=82 name=.rbd-mirror.6dfceee6-6dd3-40dc-8611-ba65a66eee86.c64d71b6-99b4-4c45-9031-f90c429916a8 size=10737418240
2016-10-25 13:51:28.626379 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=81 name=snap25 size=10737418240
2016-10-25 13:51:28.626380 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=80 name=snap24 size=10737418240
2016-10-25 13:51:28.626380 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=79 name=snap23 size=10737418240
2016-10-25 13:51:28.626381 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=78 name=snap22 size=10737418240
2016-10-25 13:51:28.626382 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=77 name=snap21 size=10737418240
2016-10-25 13:51:28.626383 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=76 name=snap20 size=10737418240
2016-10-25 13:51:28.626384 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=75 name=snap19 size=10737418240
2016-10-25 13:51:28.626384 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=74 name=snap18 size=10737418240
2016-10-25 13:51:28.626385 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=73 name=snap17 size=10737418240
2016-10-25 13:51:28.626386 7fe27b7fe700 20 librbd::image::RefreshRequest: new snapshot id=72 name=snap16 size=10737418240
2016-10-25 13:51:28.626411 7fe27b7fe700 10 librbd::image::RefreshRequest: 0x7fe264003c30 send_flush_aio
2016-10-25 13:51:28.626423 7fe27b7fe700 10 librbd::image::RefreshRequest: 0x7fe264003c30 handle_flush_aio: r=0
2016-10-25 13:51:28.626429 7fe27b7fe700 10 librbd::image::OpenRequest: handle_refresh: r=0
2016-10-25 13:51:28.626432 7fe27b7fe700 10 librbd::ImageState: 0x7fe2b73f9a40 0x7fe2b73f9a40 handle_open: r=0
2016-10-25 13:51:28.626475 7fe2ac2ded80 20 librbd: snap_list 0x7fe2b73f87b0
SNAPID NAME                                                                                      SIZE 
    72 snap16                                                                                10240 MB 
    73 snap17                                                                                10240 MB 
    74 snap18                                                                                10240 MB 
    75 snap19                                                                                10240 MB 
    76 snap20                                                                                10240 MB 
    77 snap21                                                                                10240 MB 
    78 snap22                                                                                10240 MB 
    79 snap23                                                                                10240 MB 
    80 snap24                                                                                10240 MB 
    81 snap25                                                                                10240 MB 
    82 .rbd-mirror.6dfceee6-6dd3-40dc-8611-ba65a66eee86.c64d71b6-99b4-4c45-9031-f90c429916a8 10240 MB 
    83 snap1                                                                                 10240 MB 
    84 snap2                                                                                 10240 MB 
    85 snap3                                                                                 10240 MB 
    86 snap4                                                                                 10240 MB 
    87 snap5                                                                                 10240 MB 
    88 snap6                                                                                 10240 MB 
    89 snap7                                                                                 10240 MB 
2016-10-25 13:51:28.626644 7fe2ac2ded80 20 librbd::ImageState: 0x7fe2b73f9a40 close
2016-10-25 13:51:28.626650 7fe2ac2ded80 10 librbd::ImageState: 0x7fe2b73f9a40 0x7fe2b73f9a40 send_close_unlock
2016-10-25 13:51:28.626654 7fe2ac2ded80 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_shut_down_update_watchers
2016-10-25 13:51:28.626657 7fe2ac2ded80 20 librbd::ImageState: 0x7fe2b73f9a40 shut_down_update_watchers
2016-10-25 13:51:28.626658 7fe2ac2ded80 20 librbd::ImageState: 0x7fe2b73f9b40 ImageUpdateWatchers::shut_down
2016-10-25 13:51:28.626659 7fe2ac2ded80 20 librbd::ImageState: 0x7fe2b73f9b40 ImageUpdateWatchers::shut_down: completing shut down
2016-10-25 13:51:28.626670 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_shut_down_update_watchers: r=0
2016-10-25 13:51:28.626673 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_shut_down_aio_queue
2016-10-25 13:51:28.626674 7fe27b7fe700  5 librbd::AioImageRequestWQ: shut_down: in_flight=0
2016-10-25 13:51:28.626678 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_shut_down_aio_queue: r=0
2016-10-25 13:51:28.626680 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_flush
2016-10-25 13:51:28.626683 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_flush: r=0
2016-10-25 13:51:28.626684 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_flush_readahead
2016-10-25 13:51:28.626686 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_flush_readahead: r=0
2016-10-25 13:51:28.626687 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_shut_down_cache
2016-10-25 13:51:28.626725 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_shut_down_cache: r=0
2016-10-25 13:51:28.626729 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 send_flush_op_work_queue
2016-10-25 13:51:28.626733 7fe27b7fe700 10 librbd::image::CloseRequest: 0x7fe2b73fae90 handle_flush_op_work_queue: r=0
2016-10-25 13:51:28.626750 7fe27b7fe700 10 librbd::ImageState: 0x7fe2b73f9a40 0x7fe2b73f9a40 handle_close: r=0




Note:-
Snap was removed when I disable journal on master cluster.
Comment 2 Jason Dillaman 2016-10-25 15:28:24 UTC 
Snapshots named with the prefix of ".rbd-mirror." are rbd-mirror internal snapshots.
Comment 3 Drew Harris 2017-07-11 15:13:46 UTC 
This should be fixed upstream in 3.0. Just needs validation.
Comment 7 errata-xmlrpc 2017-12-05 23:31:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387