Red Hat Bugzilla – Bug 1247135
[RFE] Ensure quality of service for sanlock io when using file-based storage
Last modified: 2018-07-18 10:41:38 EDT
Description of problem:
Sanlock is writing and reading 1MiB to every storage domain,
using 10 seconds io timeout. If writes/read to storage are too slow
for 40 seconds (4 retries based on 10 seconds timeout), sanlock will
fence the SPM (or in 3.6, any host holding the domain resource),
aborting the current vdsm operations on this host.
We are ok if storage become slow for a while, io opetaion taking many
minutes or hours will simply be slower. However, because of sanlock,
such operation running on the SPM (or any host holding a resource on
the domain) will fail when io is too slow.
We want to ensure that sanlock io have priority over other io in
For block-based storage, we use ionice with ioclass=Idle when doing
io-heavy operations like copying and converting images (e.g using dd
and qemu-img). This helps only when using cfq scheduler, but we are
using the deadline scheduler which ignore io priority. However the
deadline scheduler should serve sanlock io request on time, since
they are small. We probably should revisit the need for using ionice.
For file-based storage such as NFS and gluster, we are not doing
anything to ensure heavy io operation are not delaying sanlock io.
If seems that we can use cgrougs for this, as described in:
Version-Release number of selected component (if applicable):
David, can you take a look and add missing information?
Sahina, can you add more details regarding glusterfs?
This may be related to bug 1243935.
Vijay, is there a way to prioritize some I/O over other in gluster? Will QOS feature in gluster address this?
(In reply to Sahina Bose from comment #4)
> Vijay, is there a way to prioritize some I/O over other in gluster? Will QOS
> feature in gluster address this?
Currently we do not have any internal mechanism in Gluster to prioritize I/O based on client or application. The proposed QoS feature will provide some means but we are yet to finalize the design.
Vijay, any updates?