Description of problem:
When swift fails to check the checksum of the objects, it moves them to quarantine(/srv/node/d1/quarantined). Is there are way to recover those objects
and (re-)store back to their canonical location?
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Upload file/image to swift container
$ swift post mycontainer
$ echo "content" > file.txt
$ swift upload mycontainer file.txt
$ swift list mycontainer
2. Alter content of object
$ echo "second line" >> 712/f2a/b207d8b283f6b954f484b6c966974f2a/1459172973.66183.data
3. Object is moved to quarantine -- /srv/node/d1/quarantined/objects/
Number of objects moved to quarantine can be confirmed by the command,
$ swift-recon -q
What's the point of returning the broken object into its location if you
have just damaged it? If you succeed, you'll have a damaged object
that does not match its Etag, is all. What are you trying to accomplish?
It so happened that one of the customer re-ran overcloud deployment from different directory.
Although his deployment was successful, tripleo-overcloud-passwords changed and so does the 'path_suffix' in swift.conf. This also caused swift to unable to check the checksum of existing objects(images) which were then moved to quarantine. Actually the object are not corrupted but swift failed to verify the checksum.
It is possible to restore quarantined objects using existing tools.
Before doing it, the administrator must correct whatever issue
prompted the mass-quarantining. In particular, the hash suffix
must be restored. You must consult OSPd expert to verify that it
is sufficient to re-run it from a correct directory.
Next step is, for each object, to find out the full path where
the object resides on a storage node. This is done by examining
the metadata of the object file (with getfsattr IIRC). There is
a tool swift-object-info that can help with that. Once the proper
hash and account are known, one has to run swift-get-nodes
to reconstruct full paths to which the objects are to be restored.
Once paths are known, move the object from the quarantine directory
to its proper location. Once all objects are moved, restore
the servers with "swift-init object-server start" and verify
that the object is accessible with the old name. If that passes,
you may restart repliation and audition daemons and verify
However, in normal use it is considered extraordinary for users
to destroy seed values like this. The OSPd clearly does not have
If this safe enough to be carried out in the production environment, can you please elaborate the steps(with example commands) which I can carry out and then pass on to the customer?
(In reply to Sachin from comment #5)
> If this safe enough to be carried out in the production environment, can you
> please elaborate the steps(with example commands) which I can carry out and
> then pass on to the customer?
Sachin, please, recommend your customer re-upload images to Swift. It will be faster and safer. We'll take this case with the OSPd folks to prevent the situation described in comment#3 to happen in the future.
Your suggestion has been conveyed to the customer.
Thanx for the reply.
Closing the case.