Red Hat Bugzilla – Bug 1262480
osd: hammer: fail to start due to stray pgs after firefly->hammer upgrade
Last modified: 2017-07-30 11:21:50 EDT
Description of problem:
Notes from upstream tracker:
On Fri, Sep 11, 2015 at 8:56 PM, Sage Weil <firstname.lastname@example.org> wrote:
On Fri, 11 Sep 2015, ?? wrote:
Thank Sage Weil:
1. I delete some testing pools in the past, but is was a long
time ago (may be 2 months ago), in recently upgrade, do not
2.? ceph osd dump please see the (attachment file
3. debug osd = 20' and 'debug filestore = 20? (attachment file
This one is failing on pool 54, which has been deleted.? In this
can work around it by renaming current/54.* out of the way.
4. i install the ceph-test, but output error
ceph-kvstore-tool /ceph/data5/current/db list
Invalid argument: /ceph/data5/current/db: does not exist
(create_if_missing is false)
Sorry, I should have said current/omap, not current/db.? I'm
to see the key dump.? I'm not sure why the leveldb key for these
Yesterday I have a chat with wangrui and the reason is "infos"(legacy oid)
is missing. I'm not sure why it's missing.
Oh, I think I see what happened:
- the pg removal was aborted pre-hammer. On pre-hammer, thsi means that
load_pgs skips it here:
- we upgrade to hammer. we skip this pg (same reason), don't upgrade it,
but delete teh legacy infos object
- now we see this crash...
I think the fix is, in hammer, to bail out of peek_map_epoch if the infos
object isn't present, here
Probably we should restructure so we can return a 'fail' value
instead of a magic epoch_t meaning the same...
Version-Release number of selected component (if applicable):
Looks like sage can recreate this with modified yaml
Steps to Reproduce:
Fix that went into upstream's hammer: https://github.com/ceph/ceph/pull/5892
Based of the comment, i can think of the below mentioned scenario. Please correct me.
1. Bring the Cluster in 1.2.3 version.
2. Create a pool with 128 PGs.
3. Fill it up with some data.
4. Try to delete the pool.
5. While step 4 is in happening shutdown the OSD immediately [ The OSD must be part of the acting set for few PGs ]
So when the OSD is brought back it will still have the information of the PGs which are already being deleted from other OSD, hence leading to inconsistency after upgrade.
6. When the OSD comes back and the cluster becomes Healthy, start upgrading from 1.2.3 to 1.3.0
Can you please let me know what can be other scenario's.
Also let me know if the below makes sense.
e.g. un-mounting the OSD partition while deleting the Pool is in progress.
Ken, can you please also let us know which version to use to start upgrading from? We are planning to use 1.2.3 on RHEL 7.1 and upgrade from there to 1.3.0 async. If this is not the version to start upgrading from, then please let us know the right version.
Following test will be run downstream to verify on RH 7.1
Looking at Sage's changes to 11429.yaml, that looks like the right idea. You probably need a lot more than 128 pgs, though. The trick is that when the 'delete pool' command completes, it actually just begins an async pg deletion process. The key is to kill the osds after the deletion has begun, but before it has completed so that some of the pgs are caught in the intermediate state. You probably want to wait a bit (10s from 11429.yaml) between running the command to remove the pools and shutting down the osds (you probably want to stop all of them). I don't think un-mounting the OSD partitions is necessary.
Using the 11429.yaml directly would be better, of course!
*** Bug 1262485 has been marked as a duplicate of this bug. ***
Sorry for the confusion Federico, will close this as is gets verified in 1.3.0 async.
Verified on magna076/magna059 using 1.2.3->1.3.0 , partial logs at : http://pastebin.test.redhat.com/315887
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.