Bug 2004013
| Summary: | [DR] After performing failover mirroringStatus reports image_health: ERROR | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Pratik Surve <prsurve> |
| Component: | rook | Assignee: | Shyamsundar <srangana> |
| Status: | CLOSED ERRATA | QA Contact: | Pratik Surve <prsurve> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.9 | CC: | madam, muagarwa, ocs-bugs, odf-bz-bot, sapillai, shan, srangana |
| Target Milestone: | --- | Keywords: | TestBlocker |
| Target Release: | ODF 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-13 17:46:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 3
Santosh Pillai
2021-09-14 10:59:44 UTC
(In reply to Santosh Pillai from comment #3) > Failing to import Bootstrap token: > > From Rook Logs: > > 2021-09-08T14:57:23.080194855Z 2021-09-08 14:57:23.080053 I | cephclient: > add rbd-mirror bootstrap peer token for pool > "ocs-storagecluster-cephblockpool" > 2021-09-08T15:47:23.145091892Z 2021-09-08 15:47:23.144587 E | > ceph-block-pool-controller: failed to reconcile. failed to add ceph rbd > mirror peer: failed to import bootstrap peer token: failed to add rbd-mirror > peer token for pool "ocs-storagecluster-cephblockpool". . > 2021-09-08T14:57:23.128+0000 7ff97b7ca2c0 -1 auth: unable to find a keyring > on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.129+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.130+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > auth: unable to find a keyring on > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > bin,: (2) No such file or directory > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:47:23.136+0000 7ff97b7ca2c0 -1 > librbd::api::Mirror: peer_bootstrap_import: failed to connect to peer > cluster: (110) Connection timed out > 2021-09-08T15:47:23.145091892Z rbd: failed to import peer bootstrap token: > exit status 110 > > @Prateek Can you please the token details inside this secret > (ad6a06422d73c536059c82d8ce48b27f585fd60). Need to check the contents. Does not look like the root cause. The clusters are able to connect to each other. Tried manually bootstrapping the peer and it worked. (In reply to Santosh Pillai from comment #4) > (In reply to Santosh Pillai from comment #3) > > Failing to import Bootstrap token: > > > > From Rook Logs: > > > > 2021-09-08T14:57:23.080194855Z 2021-09-08 14:57:23.080053 I | cephclient: > > add rbd-mirror bootstrap peer token for pool > > "ocs-storagecluster-cephblockpool" > > 2021-09-08T15:47:23.145091892Z 2021-09-08 15:47:23.144587 E | > > ceph-block-pool-controller: failed to reconcile. failed to add ceph rbd > > mirror peer: failed to import bootstrap peer token: failed to add rbd-mirror > > peer token for pool "ocs-storagecluster-cephblockpool". . > > 2021-09-08T14:57:23.128+0000 7ff97b7ca2c0 -1 auth: unable to find a keyring > > on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.129+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.130+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > > auth: unable to find a keyring on > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > bin,: (2) No such file or directory > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:47:23.136+0000 7ff97b7ca2c0 -1 > > librbd::api::Mirror: peer_bootstrap_import: failed to connect to peer > > cluster: (110) Connection timed out > > 2021-09-08T15:47:23.145091892Z rbd: failed to import peer bootstrap token: > > exit status 110 > > > > @Prateek Can you please the token details inside this secret > > (ad6a06422d73c536059c82d8ce48b27f585fd60). Need to check the contents. > > Does not look like the root cause. The clusters are able to connect to each > other. Tried manually bootstrapping the peer and it worked. Santosh, it's the same error than when no ceph.conf is present but I thought you had it fixed in your PR. (In reply to Sébastien Han from comment #5) > (In reply to Santosh Pillai from comment #4) > > (In reply to Santosh Pillai from comment #3) > > > Failing to import Bootstrap token: > > > > > > From Rook Logs: > > > > > > 2021-09-08T14:57:23.080194855Z 2021-09-08 14:57:23.080053 I | cephclient: > > > add rbd-mirror bootstrap peer token for pool > > > "ocs-storagecluster-cephblockpool" > > > 2021-09-08T15:47:23.145091892Z 2021-09-08 15:47:23.144587 E | > > > ceph-block-pool-controller: failed to reconcile. failed to add ceph rbd > > > mirror peer: failed to import bootstrap peer token: failed to add rbd-mirror > > > peer token for pool "ocs-storagecluster-cephblockpool". . > > > 2021-09-08T14:57:23.128+0000 7ff97b7ca2c0 -1 auth: unable to find a keyring > > > on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.129+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T14:57:23.130+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:02:23.130+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:07:23.131+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:12:23.132+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:17:23.132+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:22:23.133+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:27:23.133+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:32:23.134+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:37:23.136+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:42:23.135+0000 7ff97b7ca2c0 -1 > > > auth: unable to find a keyring on > > > /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring. > > > bin,: (2) No such file or directory > > > 2021-09-08T15:47:23.145091892Z 2021-09-08T15:47:23.136+0000 7ff97b7ca2c0 -1 > > > librbd::api::Mirror: peer_bootstrap_import: failed to connect to peer > > > cluster: (110) Connection timed out > > > 2021-09-08T15:47:23.145091892Z rbd: failed to import peer bootstrap token: > > > exit status 110 > > > > > > @Prateek Can you please the token details inside this secret > > > (ad6a06422d73c536059c82d8ce48b27f585fd60). Need to check the contents. > > > > Does not look like the root cause. The clusters are able to connect to each > > other. Tried manually bootstrapping the peer and it worked. > > Santosh, it's the same error than when no ceph.conf is present but I thought > you had it fixed in your PR. This looks like a different error that happened when we import the bootstrap token (here https://github.com/rook/rook/blob/b3567f34a7f0ea1dace8830fbf0b1c393ede813b/pkg/daemon/ceph/client/mirror.go#L81). It does not use the ceph.conf. Just the file containing the token details of the peer cluster. The downstream ramen code was refreshed post build #125, we are looking to see when the fix [1] made it to the builds to determine if that is the root cause for the problem.
Background:
On checking Pratik's setup the error in VolumeReplication that was noted was:
{"level":"error","timestamp":"2021-09-13T20:08:25.276Z","logger":"controller-runtime.manager.controller.volumereplication","caller":"controller/controller.go:253","msg":"Reconciler error","reconciler group":"replication.storage.openshift.io","reconciler kind":"VolumeReplication","name":"busybox-pvc","namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=error"}
The mirror deamon was reporting:
debug 2021-09-13T20:10:31.714+0000 7f530277c700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5616ea6a8800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=11, local_snap_ns=[mirror state=primary (demoted), complete=1, mirror_peer_uuids=6c9c8ed3-09cb-4d62-ab2d-d87d1ca8ae09, primary_mirror_uuid=, primary_snap_id=head, last_copied_object_number=0, snap_seqs={}]
The image mirroring info was:
sh-4.4$ rbd mirror image status ocs-storagecluster-cephblockpool/csi-vol-ce53bf1f-14c3-11ec-a809-0a580a80024a
csi-vol-ce53bf1f-14c3-11ec-a809-0a580a80024a:
global_id: 737e6bfd-3c41-499d-b5bc-9b8fdd564dd0
state: up+error
description: split-brain
last_update: 2021-09-13 20:14:31
peer_sites:
name: 51f32320-86bd-40c1-a621-668a17036c48
state: up+stopped
description: local image is primary
last_update: 2021-09-13 20:14:55
The fix [1] was for VRG reconciler in Ramen to wait till VR reported Resyncing as true, to ensure we do not delete VR when image is in a split-brain. Hence checking the builds to ensure the fix is in builds post the one in use (#125)
[1] "Add more condition checks to ensure healthy secondary": https://github.com/red-hat-storage/ramen/commit/36f797df57775232199de23000727b0d0b1c2a94
> The fix [1] was for VRG reconciler in Ramen to wait till VR reported
> Resyncing as true, to ensure we do not delete VR when image is in a
> split-brain. Hence checking the builds to ensure the fix is in builds post
> the one in use (#125)
>
> [1] "Add more condition checks to ensure healthy secondary":
> https://github.com/red-hat-storage/ramen/commit/
> 36f797df57775232199de23000727b0d0b1c2a94
We are unable to do this as we force pushed to the release branch in between.
Although, before the fix the root cause seems to be the same.
Pratik requesting a retest with the latest builds to ensure this is the root cause for the same.
(In reply to Shyamsundar from comment #8) > > The fix [1] was for VRG reconciler in Ramen to wait till VR reported > > Resyncing as true, to ensure we do not delete VR when image is in a > > split-brain. Hence checking the builds to ensure the fix is in builds post > > the one in use (#125) > > > > [1] "Add more condition checks to ensure healthy secondary": > > https://github.com/red-hat-storage/ramen/commit/ > > 36f797df57775232199de23000727b0d0b1c2a94 > > We are unable to do this as we force pushed to the release branch in between. > > Although, before the fix the root cause seems to be the same. > > Pratik requesting a retest with the latest builds to ensure this is the root > cause for the same. I have started testing with the new build `4.9.0-138.ci` i will update the result when done with the testing Moved to ON_QA based on Shyam's comment. Please fail QA if it still happens. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:5086 |