Attachment 1839887 Details for Bug 2019931 – rbd-mirror.log (from primary)

rbd-mirror.log (from primary)

rbdmirror.primary.log (text/plain), 6.38 KB, created by Benamar Mekhissi on 2021-11-04 16:13:24 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Benamar Mekhissi

Created: 2021-11-04 16:13:24 UTC

Size: 6.38 KB

patch

obsolete

>debug 2021-11-02T11:42:24.538+0000 7fa9a5e0e5c0  0 set uid:gid to 167:167 (ceph:ceph)
>debug 2021-11-02T11:42:24.538+0000 7fa9a5e0e5c0  0 ceph version 16.2.0-143.el8cp (0e2c6f9639c37a03e55885fb922dc0cb1b5173cb) pacific (stable), process rbd-mirror, pid 13
>debug 2021-11-02T11:42:24.547+0000 7fa9a5e0e5c0  1 mgrc service_daemon_register rbd-mirror.129033 metadata {arch=x86_64,ceph_release=pacific,ceph_version=ceph version 16.2.0-143.el8cp (0e2c6f9639c37a03e55885fb922dc0cb1b5173cb) pacific (stable),ceph_version_short=16.2.0-143.el8cp,container_hostname=rook-ceph-rbd-mirror-a-6f959f8669-m69g9,container_image=quay.io/rhceph-dev/rhceph@sha256:81bdb1d68a68108d904d51d5df41ab74c50fc8ecb8925bb2f9b999d922bfc3a0,cpu=Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz,distro=rhel,distro_description=Red Hat Enterprise Linux 8.4 (Ootpa),distro_version=8.4,hostname=prsurve-vm-dev-s4c7k-worker-mlb4m,id=a,instance_id=129033,kernel_description=#1 SMP Tue Sep 7 07:07:31 EDT 2021,kernel_version=4.18.0-305.19.1.el8_4.x86_64,mem_cgroup_limit=2147483648,mem_swap_kb=0,mem_total_kb=32922520,os=Linux,pod_name=rook-ceph-rbd-mirror-a-6f959f8669-m69g9,pod_namespace=openshift-storage}
>debug 2021-11-02T11:42:24.609+0000 7fa9a5e0e5c0  0 rbd::mirror::PoolReplayer: 0x564bbe33db00 init_rados: reverting global config option override: keyring: /etc/ceph/keyring-store/keyring -> /etc/ceph/30952317-4f2f-4e59-a7df-6db836bca46a.client.rbd-mirror-peer.keyring,/etc/ceph/30952317-4f2f-4e59-a7df-6db836bca46a.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
>debug 2021-11-02T11:42:24.609+0000 7fa9a5e0e5c0  0 rbd::mirror::PoolReplayer: 0x564bbe33db00 init_rados: reverting global config option override: mon_host: [v2:172.31.56.87:3300,v1:172.31.56.87:6789],[v2:172.31.28.175:3300,v1:172.31.28.175:6789],[v2:172.31.213.22:3300,v1:172.31.213.22:6789] -> 
>debug 2021-11-02T11:42:24.609+0000 7fa9a5e0e5c0 -1 Errors while parsing config file!
>debug 2021-11-02T11:42:24.609+0000 7fa9a5e0e5c0 -1 can't open 30952317-4f2f-4e59-a7df-6db836bca46a.conf: (2) No such file or directory
>debug 2021-11-02T13:56:35.104+0000 7fa98c2ac700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfe2b400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=16491, local_snap_ns=[mirror state=primary (demoted), complete=1, mirror_peer_uuids=d3aaa712-4bc9-4aef-967e-56dcf3ae3585, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
>debug 2021-11-02T13:57:05.117+0000 7fa98c2ac700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bc02edf80 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.117+0000 7fa98c2ac700 -1 librbd::SnapshotRemoveRequest: 0x564bc0009e00 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T13:57:05.128+0000 7fa98c2ac700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bbfe6ae40 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.128+0000 7fa98c2ac700 -1 librbd::SnapshotRemoveRequest: 0x564bc0458180 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T13:57:05.128+0000 7fa98caad700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bc00c21c0 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.128+0000 7fa98caad700 -1 librbd::SnapshotRemoveRequest: 0x564bc0458480 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T13:57:05.129+0000 7fa98c2ac700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bc02f2f40 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.129+0000 7fa98c2ac700 -1 librbd::SnapshotRemoveRequest: 0x564bc0458000 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T13:57:05.129+0000 7fa98c2ac700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bc02ee4c0 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.129+0000 7fa98c2ac700 -1 librbd::SnapshotRemoveRequest: 0x564bc0458300 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T13:57:05.135+0000 7fa98caad700 -1 librbd::mirror::snapshot::RemoveImageStateRequest: 0x564bc0303240 handle_get_object_count: failed to read image state object: (2) No such file or directory
>debug 2021-11-02T13:57:05.135+0000 7fa98caad700 -1 librbd::SnapshotRemoveRequest: 0x564bc0458600 handle_remove_image_state: failed to remove image state: (2) No such file or directory
>debug 2021-11-02T14:38:05.128+0000 7fa9932ba700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbe345c00 get_replay_status: replay not running
>debug 2021-11-02T16:09:05.222+0000 7fa9932ba700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfdc6400 get_replay_status: replay not running
>debug 2021-11-02T16:50:05.219+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfe98c00 get_replay_status: replay not running
>debug 2021-11-02T17:54:35.261+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfef3800 get_replay_status: replay not running
>debug 2021-11-02T18:05:05.272+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfe4d000 get_replay_status: replay not running
>debug 2021-11-02T18:59:05.270+0000 7fa9932ba700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbff80800 get_replay_status: replay not running
>debug 2021-11-02T19:45:05.280+0000 7fa9932ba700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbeee8c00 get_replay_status: replay not running
>debug 2021-11-02T21:58:35.252+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bc0664000 get_replay_status: replay not running
>debug 2021-11-02T22:15:05.263+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfe4f800 get_replay_status: replay not running
>debug 2021-11-02T22:18:05.354+0000 7fa992ab9700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bbfdfec00 get_replay_status: replay not running
>debug 2021-11-02T22:19:35.256+0000 7fa9932ba700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x564bc02a1c00 get_replay_status: replay not running

Actions: View

Attachments on bug 2019931: 1839887 | 1839888 | 1839889 | 1839890 | 1839891