Bug 2172871
| Summary: | Include tar and rsync package in rook-ceph-osd pods | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | avdhoot <asagare> |
| Component: | documentation | Assignee: | Anjana Suparna Sriram <asriram> |
| Status: | NEW --- | QA Contact: | Neha Berry <nberry> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.12 | CC: | bkunal, branto, hnallurv, muagarwa, odf-bz-bot, pbyregow, srai, tmuthami, tnielsen |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
avdhoot
2023-02-23 11:19:11 UTC
cat is working only for files. In mon recovery we need to copy store.db also. Is there any way to copy directory or such files? Well, you can use ls to list the directory and copy all the contents of it, i.e. something like
for file in $(ls store.db)
do
cat store.db/$file > store.db.NEW/$file
done
but running cat/ls inside the container.
Bipin, what's the final take on this? Adding Travis for reference. The rook krew plugin [1] has a command for restoring the mon quorum, and I believe it doesn't use those packages in the run_restore_quorum() function [2]. Can you review the tool script and see if the mon recovery can be approached differently? Or even better if we can make progress on getting signoff from the support team to officially use the tool downstream. [1] https://github.com/rook/kubectl-rook-ceph#commands [2] https://github.com/rook/kubectl-rook-ceph/blob/master/kubectl-rook-ceph.sh#L253 Right, In krew plugin we don't use tar or rsync packages to recover mon or for any other operation and also restoring mon quorum is pretty much very easy, just tell the tool the good mon name one or two small details and that's it you are done no manual operation. I'll leave it to the support team now. This solution required atleast one good mon , but what if all mon in bad state? Do we have a procedure to recover the mon quorum from OSDs when all mons are in a bad state? The upstream community is very interested in this case so we can support recovery from OSDs in the krew plugin as well. https://github.com/rook/kubectl-rook-ceph/issues/74 We are not going to re-introduce these binaries/packages to the build just to support this. Moving to the docs team to document an approach here.
btw: We were able to come up with a solution for archiving this with cat/ls:
for file in $(oc exec $ROOK_OSD_POD -- ls '/tmp/rook-ceph-osd-0-5ffb7bc8f9-6l9gq/store.db/')
do
oc exec $ROOK_OSD_POD -- cat "/tmp/rook-ceph-osd-0-5ffb7bc8f9-6l9gq/store.db/$file" > /tmp/store.db/$file
done
cat is causing corruption in store.db and also it is not always consistent. I am not sure how much we can rely on cat and suggest customer to use this procedure. Also using cat/ls to copy to and from is also tedious job. For customer use, I'd probably recommend writing a robust helper script/automating this procedure somehow (maybe automating this approach in the rook krew plugin?). What store.db corruption are we talking about? This could be related to a different locales on host/pod or missing/additional whitespace somewhere. Sure. I will provide relevant changes required to install tar. |