Bug 1144852

When a disk fails in a 2-way RAID1 array, it is marked as failed in the superblock of the still-alive device and the bitmap stops being cleared.  (The bitmap is also in the metadata area along with the superblock.  Preventing clearing of the bitmap effectively tracks the changes that are made to the array while there are devices missing.)  If the machine is rebooted or the array is cycled and the failed device comes back at the same time the other device is lost, this same process of marking the failure and suspending the clearing of the bitmap will occur.  Finally, if there is another reboot or cycling of the array and all devices return, it is difficult (if not impossible) to determine which device should be the primary (i.e. the device from which to resync from).  Choosing the wrong device could potentially remove data.

This is a rare problem in the single machine case; but in an HA cluster, the problem is far more likely.  Consider a setup that has two sites - each with a server and storage.  One site is the active primary site, but data is copied to the second site via RAID1 and a storage interconnect between the sites.  If the network/storage link between the sites is broken, one site will win authority to continue working and copying stops.  The sysadmin may determine they want the other site to take over while the link is being restored.  This will cause both sides to have been active and mark the other side as failed.  When the link is restored, there is no way to determine automatically which side should be primary for the resync.  Thus, it is possible to accidentally choose the side that /wasn't/ active during the link repair and erase any changes that were made during that window.  Depending on the size of the window, the amount of data loss could be large.

A possible solution is to add a field to the superblock that indicates that the array was activated with devices missing.  Assuming this is the case, the first failure would result in the remaining device marking the other as failed, but not setting this new field (because it is already active).  If the available devices switches and the array is cycled, the now available device will mark the other as failed /and/ set the 'activated_with_missing_devs' flag.  If the array is cycled again and all the devices return, it will be obvious which side is preferred - the one the administrator chose to activate in spite of failures.  This new flag could be set on all devices if the switch happens too many times.  In this case, the admin must be asked which device should be primary before the array can be activated.

It is possible that - even with the new flag - there is no good solution.  In which case, it is simply better to detect the situation where there are conflicting failures and ask for sysadmin action at that point.  The flags solution is dependent on the assumption that the admin knowingly chooses to activate a RAID when there are failed devices and that they know the consequences.  This assumption may be flawed.  They may think it is simply the same failed device as before (they are misinformed) or the array activation may be automated (allowed activation for degraded mode).  Either of these scenarios probably invalidate the new "flag" solution.  Careful thought should be given.