Bug 1698526

Description of problem:

For creation of crush rule for ec pools its really important to understand firstn and indep .

This should be added to our Storage Strategies guide : https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/storage_strategies_guide/index#crush_rules

Mentioned here : http://docs.ceph.com/docs/master/rados/operations/crush-map-edits/#crush-map-rules

firstn versus indep

Description:	
Controls the replacement strategy CRUSH uses when items (OSDs) are marked down in the CRUSH map. If this rule is to be used with replicated pools it should be firstn and if it’s for erasure-coded pools it should be indep.

The reason has to do with how they behave when a previously-selected device fails. Let’s say you have a PG stored on OSDs 1, 2, 3, 4, 5. Then 3 goes down.

With the “firstn” mode, CRUSH simply adjusts its calculation to select 1 and 2, then selects 3 but discovers it’s down, so it retries and selects 4 and 5, and then goes on to select a new OSD 6. So the final CRUSH mapping change is 1, 2, 3, 4, 5 -> 1, 2, 4, 5, 6.

But if you’re storing an EC pool, that means you just changed the data mapped to OSDs 4, 5, and 6! So the “indep” mode attempts to not do that. You can instead expect it, when it selects the failed OSD 3, to try again and pick out 6, for a final transformation of: 1, 2, 3, 4, 5 -> 1, 2, 6, 4, 5