In the Linux kernel, the following vulnerability has been resolved:
io_uring/mpscq: disable preemption while pushing
A push makes the node the new tail, and then links it to its predecessor.
In between the queue isn't empty, but mpscq_pop() can't get at the node.
The consumers deal with that by trying again, with a cond_resched() in
the loop to give a preempted producer the chance to finish.
cond_resched() doesn't do that if the consumer has a higher priority than
the producer and both have to use the same CPU:
producer (SCHED_OTHER) consumer (SCHED_FIFO), same CPU
io_req_local_work_add()
mpscq_push()
prev = xchg(&q->tail, node)
<preempted> io_cqring_wait()
__io_run_local_work()
again:
cond_resched() // nothing to do
mpscq_pop() // NULL
// work pending, no events yet
goto again
The producer can be any task that is able to queue task_work for the
ring: an io-wq worker, or somebody who sends a message with
IORING_OP_MSG_RING. The consumer spins, holding ->uring_lock, until the
scheduler takes pity on the fair class, which takes about a second with
the default settings of the fair server, and forever without it. The
loops in tctx_task_work_run() and io_cancel_local_task_work() have the
same problem.
Don't allow preemption between the two steps. That bounds the time the
window is open on the CPU of the producer by what interrupts take, and a
consumer that waits for that is on another CPU. The cost is a
preempt_disable()/preempt_enable() pair per push.
In the Linux kernel, the following vulnerability has been resolved: io_uring/mpscq: disable preemption while pushing A push makes the node the new tail, and then links it to its predecessor. In between the queue isn't empty, but mpscq_pop() can't get at the node. The consumers deal with that by trying again, with a cond_resched() in the loop to give a preempted producer the chance to finish. cond_resched() doesn't do that if the consumer has a higher priority than the producer and both have to use the same CPU: producer (SCHED_OTHER) consumer (SCHED_FIFO), same CPU io_req_local_work_add() mpscq_push() prev = xchg(&q->tail, node) <preempted> io_cqring_wait() __io_run_local_work() again: cond_resched() // nothing to do mpscq_pop() // NULL // work pending, no events yet goto again The producer can be any task that is able to queue task_work for the ring: an io-wq worker, or somebody who sends a message with IORING_OP_MSG_RING. The consumer spins, holding ->uring_lock, until the scheduler takes pity on the fair class, which takes about a second with the default settings of the fair server, and forever without it. The loops in tctx_task_work_run() and io_cancel_local_task_work() have the same problem. Don't allow preemption between the two steps. That bounds the time the window is open on the CPU of the producer by what interrupts take, and a consumer that waits for that is on another CPU. The cost is a preempt_disable()/preempt_enable() pair per push.