Description of problem (please be detailed as possible and provide log snippests): oc cp and oc rsync command is failing with below error- WARNING: cannot use rsync: rsync not available in container WARNING: cannot use tar: tar not available in container error: No available strategies to copy. The tar utility is missing in ODF 4.12 from rook-ceph-osd pods. Version of all relevant components (if applicable): ODF 4.12 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
cat is working only for files. In mon recovery we need to copy store.db also. Is there any way to copy directory or such files?
Well, you can use ls to list the directory and copy all the contents of it, i.e. something like for file in $(ls store.db) do cat store.db/$file > store.db.NEW/$file done but running cat/ls inside the container.
Bipin, what's the final take on this? Adding Travis for reference.
The rook krew plugin [1] has a command for restoring the mon quorum, and I believe it doesn't use those packages in the run_restore_quorum() function [2]. Can you review the tool script and see if the mon recovery can be approached differently? Or even better if we can make progress on getting signoff from the support team to officially use the tool downstream. [1] https://github.com/rook/kubectl-rook-ceph#commands [2] https://github.com/rook/kubectl-rook-ceph/blob/master/kubectl-rook-ceph.sh#L253
Right, In krew plugin we don't use tar or rsync packages to recover mon or for any other operation and also restoring mon quorum is pretty much very easy, just tell the tool the good mon name one or two small details and that's it you are done no manual operation. I'll leave it to the support team now.
This solution required atleast one good mon , but what if all mon in bad state?
Do we have a procedure to recover the mon quorum from OSDs when all mons are in a bad state? The upstream community is very interested in this case so we can support recovery from OSDs in the krew plugin as well. https://github.com/rook/kubectl-rook-ceph/issues/74
We are not going to re-introduce these binaries/packages to the build just to support this. Moving to the docs team to document an approach here. btw: We were able to come up with a solution for archiving this with cat/ls: for file in $(oc exec $ROOK_OSD_POD -- ls '/tmp/rook-ceph-osd-0-5ffb7bc8f9-6l9gq/store.db/') do oc exec $ROOK_OSD_POD -- cat "/tmp/rook-ceph-osd-0-5ffb7bc8f9-6l9gq/store.db/$file" > /tmp/store.db/$file done
cat is causing corruption in store.db and also it is not always consistent. I am not sure how much we can rely on cat and suggest customer to use this procedure. Also using cat/ls to copy to and from is also tedious job.
For customer use, I'd probably recommend writing a robust helper script/automating this procedure somehow (maybe automating this approach in the rook krew plugin?). What store.db corruption are we talking about? This could be related to a different locales on host/pod or missing/additional whitespace somewhere.
Sure. I will provide relevant changes required to install tar.