Bug 1293975

Summary: RFE: Operational Blockers for BDS Nodes in QEMU block layer
Product: Red Hat Enterprise Linux 7 Reporter: Jeff Cody <jcody>
Component: qemu-kvm-rhevAssignee: Kevin Wolf <kwolf>
Status: CLOSED ERRATA QA Contact: Qianqian Zhu <qizhu>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.3CC: chayang, huding, juzhang, kwolf, michen, mrezanin, mtessun, qzhang, virt-maint, xfu, xianwang, zhengtli
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.4   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 23:29:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Cody 2015-12-23 20:57:31 UTC
To provide safe access to BDS nodes for various simultaneous BDS graph operations (such as those performed by a block job, like block-commit and block-stream), we need a more advanced operational blocker system in place.

There is a proposed RFC upstream, that contains more details:
https://lists.gnu.org/archive/html/qemu-block/2015-12/msg00257.html

Comment 2 Kevin Wolf 2017-04-03 13:07:07 UTC
The basic infrastructure for the new op blocker system has arrived in time for
2.9, so their restrictions are in effect now. There were several bugs before rc1,
so I'm setting "Fixed in" to qemu 2.9.0-rc1 now.

We need to test that the new op blockers don't block any operations that we
actually want to be able to perform.

Comment 4 Qianqian Zhu 2017-05-04 06:05:57 UTC
Hi Kevin,

QE is not sure how to verify this bz, could you help indicate? 

Thanks,
Qianqian

Comment 5 Kevin Wolf 2017-05-04 10:27:48 UTC
This is infrastructure, so we can't test it directly, but only by testing its
users. There are many users and many potential incompatibilities between them
that op blocker should detect, but we can pick out a simple one: Multiple guest
devices accessing the same image.

First of all, create a block node with a node-name:

    -drive if=none,file=/tmp/test.qcow2,node-name=disk

Then add two virtio-blk devices that use the same node as their backend:

    -device virtio-blk-pci,drive=disk
    -device virtio-blk-pci,drive=disk

This should result in an error message like the following because virtio-blk
requires by default that nobody else is writing to the image:

qemu-system-x86_64: -device virtio-blk-pci,drive=disk: Conflicts with use by
/machine/peripheral-anon/device[0]/virtio-backend as 'root', which does not
allow 'write' on disk

You can modify the -device command line so that a virtio-blk device allows
shared writers:

    -device virtio-blk-pci,drive=disk,share-rw=on
    -device virtio-blk-pci,drive=disk,share-rw=on

This should succeed.

Instead of a second virtio-blk device, you can also test other things that
would make changes to the image, e.g. using it as a block job target (streaming
should work though, because it doesn't change the disk content) or starting a
writable NBD server for the node.

Does this give you some ideas how to test the feature?

Comment 6 Qianqian Zhu 2017-05-05 09:38:33 UTC
Thanks Kevin.

With qemu-kvm-rhev-2.9.0-2.el7.x86_64, I have tested the second virtio-blk device scenario:
# /usr/libexec/qemu-kvm -name rhel7_4-9343 -m 1G -smp 2 -drive file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2,format=qcow2,if=none,cache=none,media=disk,werror=stop,rerror=stop,node-name=disk1 -device virtio-blk-pci,drive=disk1,id=virtio-blk-0 -device virtio-blk-pci,drive=disk1,id=virtio-blk-1 -monitor stdio
QEMU 2.9.0 monitor - type 'help' for more information

Qemu report error:
(qemu) qemu-kvm: -device virtio-blk-pci,drive=disk1,id=virtio-blk-1: Conflicts with use by /machine/peripheral/virtio-blk-0/virtio-backend as 'root', which does not allow 'write' on disk1

With share-rw=on, it is launched successful:

#  /usr/libexec/qemu-kvm -name rhel7_4-9343 -m 1G -smp 2 -drive file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2,format=qcow2,if=none,cache=none,media=disk,werror=stop,rerror=stop,node-name=disk1 -device virtio-blk-pci,drive=disk1,id=virtio-blk-0,share-rw=on -device virtio-blk-pci,drive=disk1,id=virtio-blk-1,share-rw=on -monitor stdio

I did not find a way to test it as a block job target, since all block job target is specified by file name instead of node name. I have tried specify a file already used by a node as block job target, it will be overwrote.

But from my understanding, this scheme should handle the conflict of simultaneous operations to a node, forbidden improper operations or execute them sequentially. 
Based on my test mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1367731#c2, this should works as expected.

Kevin, do you think it can be verified based on above result? Thanks.

Comment 7 Qianqian Zhu 2017-05-05 09:51:42 UTC
Please help check https://bugzilla.redhat.com/show_bug.cgi?id=1447184, if this regression is introduced by this new scheme, since the error message of this issue is "Conflicts with use by drive-scsi-disk0 as 'backing', which does not allow 'write' on #block198"

Comment 8 Kevin Wolf 2017-05-05 15:36:00 UTC
(In reply to Qianqian Zhu from comment #6)
> I did not find a way to test it as a block job target, since all block job
> target is specified by file name instead of node name. I have tried specify
> a file already used by a node as block job target, it will be overwrote.

You are right that the older block job commands (drive-mirror and drive-backup)
use a filename for the target. But there are newer commands that take a node
name instead: blockdev-mirror and blockdev-backup.

> But from my understanding, this scheme should handle the conflict of
> simultaneous operations to a node, forbidden improper operations or execute
> them sequentially.

Correct.

> Kevin, do you think it can be verified based on above result? Thanks.

You can still try the block job ones, but other than that I would say yes.

(In reply to Qianqian Zhu from comment #7)
> Please help check https://bugzilla.redhat.com/show_bug.cgi?id=1447184, if
> this regression is introduced by this new scheme, since the error message of
> this issue is "Conflicts with use by drive-scsi-disk0 as 'backing', which
> does not allow 'write' on #block198"

The error message is definitely created by the op blockers. It may be a problem
with the op blockers, or a separate bug that was only detected by op blockers.
Let's investigate this in the other BZ.

Comment 9 Qianqian Zhu 2017-05-08 02:53:29 UTC
Test blockdev-backup and blockdev-mirror conflict scenario:
Steps:
1. Launch guest with two block device in use:
#  /usr/libexec/qemu-kvm -name rhel7_4-9343 -m 1G -smp 2  -drive file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2,format=qcow2,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive-0 -device virtio-blk-pci,drive=drive-0,id=virtio-blk-0,bootindex=0 -drive file=/home/test,format=qcow2,if=none,cache=none,aio=native,id=drive-virtio-blk1,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-blk1,id=virtio-blk1,bus=pci.0,addr=0x14,serial="QEMU-DISK2" -monitor stdio -qmp tcp:0:5555,server,nowait -vnc :3 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=70:e2:84:14:0e:15

2. Try to blockdev-backup and blockdev-mirror from one to another:
{   "execute": "blockdev-backup", "arguments": {"device": "drive-0", "target": "drive-virtio-blk1","sync": "full"}}
{   "execute": "blockdev-mirror", "arguments": {"device": "drive-0", "target": "drive-virtio-blk1","sync": "full"}}

Result:
Actions are forbidden with error message prompt:
{"error": {"class": "GenericError", "desc": "Conflicts with use by drive-virtio-blk1 as 'root', which does not allow 'write' on #block322"}}

Comment 10 Qianqian Zhu 2017-05-08 02:54:28 UTC
Per comment 6, comment 8 and comment 9, moving to VERIFIED.

Comment 12 errata-xmlrpc 2017-08-01 23:29:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 13 errata-xmlrpc 2017-08-02 01:07:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 14 errata-xmlrpc 2017-08-02 01:59:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 15 errata-xmlrpc 2017-08-02 02:40:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 16 errata-xmlrpc 2017-08-02 03:04:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 17 errata-xmlrpc 2017-08-02 03:24:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392