>debug 2022-11-24T02:21:51.096+0000 7f4806ed4700 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55bba0a3cf70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.096+0000 7f4806ed4700 -1 librbd::io::AioCompletion: 0x55bb9fbccdc0 fail: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.097+0000 7f48076d5700  0 [rbd_support DEBUG root] CreateSnapshotRequests.handle_create_snapshot for 1//10bf6aad31987: r=-108, snap_id=None
debug 2022-11-24T02:21:51.097+0000 7f48076d5700  0 [rbd_support ERROR root] error when creating snapshot for 1//10bf6aad31987: -108
debug 2022-11-24T02:21:51.097+0000 7f48076d5700  0 [rbd_support DEBUG root] CreateSnapshotRequests.close_image 1//10bf6aad31987
debug 2022-11-24T02:21:51.097+0000 7f47f8d78700  0 [rbd_support CRITICAL root] Fatal runtime error: [errno 108] RBD connection was shutdown (error opening image b'10bf66fd5c311' at snapshot None)
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rbd_support/mirror_snapshot_schedule.py", line 327, in run
    refresh_delay = self.refresh_images()
  File "/usr/share/ceph/mgr/rbd_support/mirror_snapshot_schedule.py", line 363, in refresh_images
    self.load_schedules()
  File "/usr/share/ceph/mgr/rbd_support/mirror_snapshot_schedule.py", line 352, in load_schedules
    schedules.load(namespace_validator, image_validator)
  File "/usr/share/ceph/mgr/rbd_support/schedule.py", line 375, in load
    image_validator)
  File "/usr/share/ceph/mgr/rbd_support/schedule.py", line 405, in load_from_pool
    image_validator)
  File "/usr/share/ceph/mgr/rbd_support/schedule.py", line 191, in from_id
    read_only=True) as image:
     File "rbd.pyx", line 2894, in rbd.Image.__init__
rbd.ConnectionShutdown: [errno 108] RBD connection was shutdown (error opening image b'10bf66fd5c311' at snapshot None)

debug 2022-11-24T02:21:51.097+0000 7f4806ed4700 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.097+0000 7f4806ed4700 -1 librbd::io::AioCompletion: 0x55bb9fb7c580 fail: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.097+0000 7f4806ed4700 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55bba046ac80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.098+0000 7f48076d5700 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.099+0000 7f48076d5700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.100+0000 7f4806ed4700 -1 librbd::ImageState: 0x55bba0a7e580 failed to open image: (108) Cannot send after transport endpoint shutdown
debug 2022-11-24T02:21:51.100+0000 7f48243de700  0 [rbd_support CRITICAL root] Fatal runtime error: [errno 108] RBD connection was shutdown (error opening image b'csi-vol-9384ccfd-2c93-4959-a2de-b692e23052e9' at snapshot None)
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rbd_support/module.py", line 174, in handle_command
    inbuf, prefix[29:], cmd)
  File "/usr/share/ceph/mgr/rbd_support/mirror_snapshot_schedule.py", line 579, in handle_command
    image_validator)
  File "/usr/share/ceph/mgr/rbd_support/schedule.py", line 130, in from_name
    read_only=True) as image:



@Elvir. I would suggest running the test with rbd CLI commands. Looks like you might be hitting any RBD bug.

@Ilya any idea when there is a panic in the mgr pod? looks like its connection/network issue.