With more users adopting KVM-based solutions (RHEV and OSP), we've
been receiving more escalations from GSS, as expected.

And one recurrent kind of escalation which usually arrives with a
very high priority is some sort of image corruption. Based on
recent history, these corruptions are not caused by QEMU, but the
burden of the investigation is usually on us.

I even remember hearing something like "we've seen corruptions in
the field" from somebody the other day, even though it's been a
while since the last time we fixed a real corruption bug in
RHEL/QEMU. In summary: when a corruption happens, we look
bad, even if it's not our fault.

Some recent examples:

 - The usual image corruption resulting from using qemu-img in an
   open image
   https://bugzilla.redhat.com/show_bug.cgi?id=1277167

 - os-probe (apparently from grub) remounts host partitions
   (read-only) when looking for additional OSes installed in the
   system. In case of XFS partitions, the journal gets replayed,
   resulting in corruption if they're being used by a guest
   https://bugzilla.redhat.com/show_bug.cgi?id=1304321

 - A hardware inventory program (from the customer) that decided
   it was a good idea to probe entries from /dev by sending a few
   bytes to each suspicious device (looking for printers, for
   example)
   https://bugzilla.redhat.com/show_bug.cgi?id=1241510

 - Strange qcow2 corruption in a setup with Ceph hosting qcow2
   images (after weeks of debugging, it turned out to be a race
   in Ceph's cache implementation, but it took us a long time just
   to exclude the possibility that the image was being open by
   a different process)
   https://bugzilla.redhat.com/show_bug.cgi?id=1308433

What all these problems have in common is that they took a long
time to debug, resulting in frustration for everybody: customers,
GSS and us.

So the question is: what can we do to improve the situation? Not
just prevent these issues from happening, but also improve the
tools we have to audit the problem?

Libvirt already does some locking and there's work upstream to add image locking to QEMU, thus preventing multiple QEMU or qemu-img from opening an image in use. But that is not enough. As can be seen im the examples above, we also need a mechanism to detect when other programs write to a block device (or image) used by QEMU.

From recent discussions, here are a few ideas:

    * libvirt -- permissions + SELinux to prevent users from
      accessing the block device in use

    * inotify: log a warning (or taint the domain) if the block device
      is opened by a different process
      * Use IN_CLOSE_WRITE event to detect when another process
        had the file open for writing? What about opening?

    * Integrate with auditctl (see https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Defining_Audit_Rules_and_Controls.html)

    * Raise requirements for API improvements in Ceph and Gluster

    * Figure out something for NFS hosted images

    * Have all I/O against network storage in a QEMU native backend
      not kernel (avoiding any issues with device nodes appearing in
      /dev on the host)

I'm assuming the best place for this kind of protection is in libvirt. Maybe it should be handled inside QEMU instead (although there are cases where the changes would have to be in QEMU). Most likely, a proper solution would involve changes in both QEMU and libvirt.