Bug 1856960

Cause: There was not enough checks on BlueFS log replay log size. When OSD was not processing any external requests, it still was periodically sending a small update to RocksDB. This translated to appending some to BlueFS log. Note: any actual OP on OSD would have triggered log compaction. Consequence: BlueFS log grows so large that it can no longer be read. It remained unnoticed until OSD restart. Fix: Once error condition is reached, OSD is unable to boot. Heuristic procedure has been created that attempts to find on device missing parts of log. It is enabled when "bluefs_replay_recovery=true" is set. Because it is only heuristic solution, fsck is necessary to check if process was successful. In normal mode, BlueFS compacts log right after bootup. To prevent this compaction "bluefs_replay_recovery_disable_compact=true" should be used until *fsck* returns success. So, fix procedure is 2 steps: 1) CHECK ceph-bluestore-tool -l /proc/self/fd/1 --log-level 5 --path *osd path* fsck --debug_bluefs=5/5 --bluefs_replay_recovery=true --bluefs_replay_recovery_disable_compact=true 2) ACTUAL FIX ceph-bluestore-tool -l /proc/self/fd/1 --log-level 5 --path *osd path* fsck --debug_bluefs=5/5 --bluefs_replay_recovery=true Result: Now OSD can boot up.