Bug 1623986

Summary: block-commit can't be used with -blockdev
Product: Red Hat Enterprise Linux 7 Reporter: Peter Krempa <pkrempa>
Component: qemu-kvm-rhevAssignee: Kevin Wolf <kwolf>
Status: CLOSED ERRATA QA Contact: aihua liang <aliang>
Severity: high Docs Contact:
Priority: high    
Version: 7.5CC: aliang, areis, chayang, coli, guillaume.pavese, jsuchane, juzhang, kwolf, mtessun, ngu, pkrempa, virt-maint
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-20.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1644996 1650175 (view as bug list) Environment:
Last Closed: 2019-08-22 09:18:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 760547, 1636224, 1644996, 1650175, 1651787    

Description Peter Krempa 2018-08-30 15:31:18 UTC 
Description of problem:
When using a node-name for the @device argument qemu does not reopen the appropriate files (@base) read-write and thus block-commit is not possible.

Note that marking the backing chain read-write would mean that parts of the backing chain can't be shared due to image locking.


Version-Release number of selected component (if applicable):
Current upstream.

{"execute":"block-commit",
 "arguments": { "device":"libvirt-3-format",
                "job-id":"libvirt-3-format",
                "top-node":"libvirt-8-format",
                "base-node":"libvirt-9-format",
                "auto-finalize":true,
                "auto-dismiss":false},
 "id":"libvirt-16"}

{"id":"libvirt-16",
 "error":{"class":"GenericError",
          "desc":"Block node is read-only"}}

I'm pointing into the backing chain so the files are declared as read-only.

It works just-fine if I open them as read-write with
-blockdev/blockdev-add but that obviously is not correct as you can't
then share parts of the backing chain with other VMs due to image
locking.

libvirt-3-format is read-write and all other node names are readonly in
the above example.

The same also happens when using filenames:

{"execute":"block-commit",
 "arguments" : {"device":"libvirt-3-format",
                "job-id":"libvirt-3-format",
                "top":"/var/lib/libvirt/images/rhel7.3.1483615252",
                "base":"/var/lib/libvirt/images/rhel7.3.1483605924",
                "auto-finalize":true,
                "auto-dismiss":false},
 "id":"libvirt-13"}

{"id":"libvirt-13","error":{"class":"GenericError","desc":"Block node is 
read-only"}}


When I use the drive alias rather than the node-name for the 'device'
argument it works as expected.

{"execute":"block-commit",
 "arguments": { "device":"drive-virtio-disk0",
                "job-id":"drive-virtio-disk0",
                "top":"/var/lib/libvirt/images/rhel7.3.1483536402",
                "base":"/var/lib/libvirt/images/rhel7.3.1483545313"},
 "id":"libvirt-18"}


https://lists.gnu.org/archive/html/qemu-block/2018-08/msg00888.html
Comment 5 Kevin Wolf 2018-09-05 08:58:34 UTC 
*** Bug 1625257 has been marked as a duplicate of this bug. ***
Comment 20 Miroslav Rezanina 2018-12-06 12:40:02 UTC 
Fix included in qemu-kvm-rhev-2.12.0-20.el7
Comment 33 aihua liang 2019-01-18 08:45:01 UTC 
Test env:
  kernel version:3.10.0-957.1.3.el7.x86_64
  qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-21.el7.x86_64

For reproduce: 
 test step:
  step1.Create 3 images offline:
    #qemu-img create -f qcow2 base 1G
    #qemu-img create -f qcow2 /home/atest/sn1 1G
    #qemu-img create -f qcow2 /home/atest/sn2 1G

  step2.Start qemu with snapshot chain: base->sn1->sn2
     /usr/libexec/qemu-kvm \
    -blockdev driver=file,node-name=file_base,filename=/home/atest/base,read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=base,read-only=on \
    -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,read-only=on \
    -blockdev driver=qcow2,file=file_sn1,node-name=sn1,read-only=on,backing=base \
    -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
    -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \
    -qmp tcp:0:5000,server,nowait \
    -monitor stdio \
    -vnc :0 \

  step3.Do block mirror: sn1->base
    {'execute': 'block-commit', 'arguments': { 'device':'sn2','base-node':'base','top-node':'sn1','job-id':'j1'}}
  
 test result:
  after step3:
    block-commit failed with following info:
    {"timestamp": {"seconds": 1547605428, "microseconds": 437870}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
    {"timestamp": {"seconds": 1547605428, "microseconds": 438245}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}
    {"error": {"class": "GenericError", "desc": "Block node is read-only"}}


For verify:
  1.backend:localfs
    test steps:
     step1: same with reproduce step1

     step2: Start qemu with snapshot chain: base->sn1->sn2
     /usr/libexec/qemu-kvm \
      -blockdev driver=file,node-name=file_base,filename=/home/atest/base,auto-read-only=on \
      -blockdev driver=qcow2,file=file_base,node-name=base,auto-read-only=on \
      -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
      -blockdev driver=qcow2,file=file_sn1,node-name=sn1,read-only=on,backing=base \
      -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
      -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \
      -qmp tcp:0:5000,server,nowait \
      -monitor stdio \
      -vnc :0 \

     step3: same with reproduce step3
   
   test result:
     block commit executed successfully.
       {"timestamp": {"seconds": 1547606177, "microseconds": 966791}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 967306}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
       {"return": {}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 982351}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 982409}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 982900}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "commit"}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 982964}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
       {"timestamp": {"seconds": 1547606177, "microseconds": 982999}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}

 2.backend:rbd
    test steps:
     step1: create 3 images
      # qemu-img create -f raw rbd:rbd/sn1_r:mon_host=10.66.144.31 1G
      # qemu-img create -f qcow2 /home/atest/sn1 1G
      # qemu-img create -f qcow2 /home/atest/sn2 1G

     step2: Start qemu with snapshot chain: base->sn1->sn2
       -blockdev driver=rbd,node-name=file_base,pool=rbd,image=base,server.0.host=10.66.144.31,server.0.port=6789,auto-read-only=on \
       -blockdev driver=raw,file=file_base,node-name=base,auto-read-only=on \
       -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
       -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
       -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
       -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \
       

     step3: same with reproduce step3
   
   test result:
     block commit executed successfully.

  3.backend:ndb
    test steps:
      step1: create 3 images
       (nbd server)#qemu-img create -f raw base 1G; qemu-nbd -f raw base -p 9000 -t
       # qemu-img create -f qcow2 /home/atest/sn1 1G
       # qemu-img create -f qcow2 /home/atest/sn2 1G
      
      step2: Start qemu with snapshot chain: base->sn1->sn2
       -blockdev driver=nbd,node-name=file_base,server.host=10.73.73.83,server.port=9000,server.type=inet,auto-read-only=on \
       -blockdev driver=raw,file=file_base,node-name=base,auto-read-only=on \
       -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
       -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
       -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
       -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \

      step3: same with reproduce step3

    test result:
      block commit executed successfully.

  4.backend:gluster
     test steps:
      step1: create 3 images
       # qemu-img create -f qcow2 gluster://gluster-virt-qe-01.lab.eng.pek2.redhat.com/vol0/base 1G
       # qemu-img create -f qcow2 /home/atest/sn1 1G
       # qemu-img create -f qcow2 /home/atest/sn2 1G
      
      step2: Start qemu with snapshot chain: base->sn1->sn2
       -blockdev driver=gluster,node-name=file_base,server.0.type=inet,server.0.host=gluster-virt-qe-01.lab.eng.pek2.redhat.com,server.0.port=24007,volume=vol0,path=base,auto-read-only=on \
       -blockdev driver=qcow2,file=file_base,node-name=base,auto-read-only=on \
       -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
       -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
       -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
       -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \

      step3: same with reproduce step3

    test result:
      block commit executed successfully.
  
  5.backend:iscsi
     step1: create 2 images
       # qemu-img create -f qcow2 gluster://gluster-virt-qe-01.lab.eng.pek2.redhat.com/vol0/base 1G
       # qemu-img create -f qcow2 /home/atest/sn1 1G
       # qemu-img create -f qcow2 /home/atest/sn2 1G
      
      step2: Start qemu with snapshot chain: base->sn1->sn2
       -blockdev driver=iscsi,node-name=file_base,transport=tcp,portal=10.73.73.83,target=iqn.2018-02.com.example:t1,lun=0,auto-read-only=on \
       -blockdev driver=raw,file=file_base,node-name=base,auto-read-only=on \
       -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
       -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
       -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
       -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \

      step3: same with reproduce step3

    test result:
      block commit executed successfully.
  6.backend:curl
      driver: https
        step1: create 3 images
         (https server)# qemu-img create -f qcow2 base 1G
         # qemu-img create -f qcow2 /home/atest/sn1 1G
         # qemu-img create -f qcow2 /home/atest/sn2 1G
      
        step2: start qemu with snapshot chain: base->sn1->sn2
         -blockdev driver=iscsi,node-name=file_base,driver=https,sslverify=off,url=https://10.73.224.153/base,read-only=on,auto-read-only=on \
         -blockdev driver=raw,file=file_base,node-name=base,auto-read-only=on \
         -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
         -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
         -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
         -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \

    test result:
         qemu-kvm: -blockdev driver=https,node-name=file_base,sslverify=off,url=https://10.73.224.153/base,auto-read-only=on: Driver 'https' can only be used for read-only devices

     driver: http
      test steps:
        step1: same as step1 above
        step2: start qemu with snapshot chain: base->sn1->sn2
         -blockdev driver=iscsi,node-name=file_base,driver=http,url=https://10.73.224.153/base,read-only=on,auto-read-only=on \
         -blockdev driver=raw,file=file_base,node-name=base,auto-read-only=on \
         -blockdev driver=file,node-name=file_sn1,filename=/home/atest/sn1,auto-read-only=on \
         -blockdev driver=qcow2,file=file_sn1,node-name=sn1,auto-read-only=on,backing=base \
         -blockdev driver=file,node-name=file_sn2,filename=/home/atest/sn2 \
         -blockdev driver=qcow2,file=file_sn2,node-name=sn2,backing=sn1 \
    
      test result:
         qemu start failed with info:
           qemu-kvm: -blockdev driver=http,node-name=file_base,url=http://10.73.224.153/base,read-only=on: Driver 'http' is not whitelisted
         But check img info via http success:
           qemu-img info 'json:{"file.driver":"http", "file.url":"http://10.73.224.153/base"}
           > '
             image: json:{"driver": "qcow2", "file": {"url": "http://10.73.224.153/base", "driver": "http"}}
             file format: qcow2
             virtual size: 1.0G (1073741824 bytes)
             disk size: unavailable
             cluster_size: 65536
             Format specific information:
               compat: 1.1
               lazy refcounts: false
               refcount bits: 16
               corrupt: false
Comment 34 aihua liang 2019-01-21 09:29:31 UTC 
Hi, kevin

  New issue need to confirm:
    Failed to do qemu-io write on a node-name:
     (qemu) qemu-io drive_image1 "write -P 0xa 0 512"
       Conflicts with use by /machine/peripheral/image1/virtio-backend as 'root', which does not allow 'write' on drive_image1

    It seems another read-only issue, i'm not sure it has some relationship with this bug.

    Qemu cmds to start:
    -blockdev node-name=file_node,driver=file,filename=/home/kvm_autotest_root/images/rhel76-64-virtio.qcow2,auto-read-only=on \
    -blockdev node-name=drive_image1,file=file_node,driver=qcow2,auto-read-only=on \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \


    Note: 
      1. Whether using auto-read-only=on or not, the issue also exit.
      2. When use a alias-name for -drive id=xxx, it works fine.
          (qemu) qemu-io drive_image1 "write -P 0xaa 0 512"
wrote 512/512 bytes at offset 0
512 bytes, 1 ops; 0.0001 sec (4.246 MiB/sec and 8695.6522 ops/sec)
           Qemu cmds to start:
             -drive if=none,id=drive_image1,file=/home/kvm_autotest_root/images/rhel76-64-virtio.qcow2,format=qcow2 \
             -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \


BR,
aliang
Comment 35 Kevin Wolf 2019-01-21 13:56:50 UTC 
virtio-blk-pci does not like other writers on its image unless you set share-rw=on. HMP 'qemu-io <node-name>' is a separate user, so you can't get the write permission as long as this virtio-blk device is still attached. With 'qemu-io <drive-id>' the request instead looks as if it came from the virtio-blk itself (the same BlockBackend is used for request submissions), so the checks don't apply.
Comment 36 aihua liang 2019-01-22 02:25:50 UTC 
(In reply to Kevin Wolf from comment #35)
> virtio-blk-pci does not like other writers on its image unless you set
> share-rw=on. HMP 'qemu-io <node-name>' is a separate user, so you can't get
> the write permission as long as this virtio-blk device is still attached.
> With 'qemu-io <drive-id>' the request instead looks as if it came from the
> virtio-blk itself (the same BlockBackend is used for request submissions),
> so the checks don't apply.

Yes, test with share-rw=on, it works fine, and thank you, kevin.
Comment 38 aihua liang 2019-01-22 02:36:06 UTC 
According to test result in comment 33 and comment 37, no new bugs introduced, so set bug's status to "Verified".
Comment 40 errata-xmlrpc 2019-08-22 09:18:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553