Bug 1189998

Summary: Active commit does not support on rbd based disk
Product: Red Hat Enterprise Linux 7 Reporter: Yang Yang <yanyang>
Component: qemu-kvm-rhevAssignee: Jeff Cody <jcody>
Status: CLOSED ERRATA QA Contact: Suqin Huang <shuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: chayang, coli, dyuan, hhuang, huding, jcody, juzhang, michen, mzhan, rhbug, ruben, shuang, virt-maint, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1200726 (view as bug list) Environment:
Last Closed: 2017-08-01 23:27:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1200726    

Description Yang Yang 2015-02-06 03:16:59 UTC
Description of problem:
Inactive blockcommit with rbd based disk works well. But active blockcommit does NOT work.

Version-Release number of selected component (if applicable):
libvirt-1.2.8-16.el7.x86_64
qemu-kvm-rhev-2.1.2-23.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.define/start a guest with the following xml

<disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='rbd' name='libvirt-pool/rbd1.img'>
      <config file='/etc/ceph.conf'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>

virsh start vm3
Domain vm3 started

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 24    vm3                            running

2. check qemu command line
# ps -ef|grep qemu
qemu     19364     1 42 11:04 ?        00:00:15 /usr/libexec/qemu-kvm -name vm3 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu SandyBridge -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 00469895-ad03-4033-96c5-19c5da2f3ee3 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
......

3. create 3 external snapshots
# for i in {1..3}; do virsh snapshot-create-as vm3 s$i --disk-only --diskspec vda,file=/tmp/vm1.s$i; done
Domain snapshot s1 created
Domain snapshot s2 created
Domain snapshot s3 created

# qemu-img info /tmp/vm1.s3 --backing-chain
image: /tmp/vm1.s3
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 1.1M
cluster_size: 65536
backing file: /tmp/vm1.s2
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /tmp/vm1.s2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536
backing file: /tmp/vm1.s1
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: /tmp/vm1.s1
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536
backing file: rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf
backing file format: raw
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf
file format: raw
virtual size: 5.0G (5368709120 bytes)
disk size: unavailable
cluster_size: 4194304

4. check domain xml
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp/vm1.s3'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/tmp/vm1.s2'/>
        <backingStore type='file' index='2'>
          <format type='qcow2'/>
          <source file='/tmp/vm1.s1'/>
          <backingStore type='network' index='3'>
            <format type='raw'/>
            <source protocol='rbd' name='libvirt-pool/rbd1.img'>
              <config file='/etc/ceph.conf'/>
            </source>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>

5. do active commit
# virsh blockcommit vm3 vda --wait --verbose --active
error: internal error: unable to execute QEMU command 'block-commit': 
Block format 'rbd' used by device '' does not support feature 'reopening of file'

6. do blockcommit from middle to base
# virsh blockcommit vm3 vda --wait --verbose --top /tmp/vm1.s2
Block Commit: [100 %]
Now in synchronized phase

# qemu-img info /tmp/vm1.s3 --backing-chain
image: /tmp/vm1.s3
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 1.1M
cluster_size: 65536
backing file: rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf
backing file format: raw
Format specific information:
    compat: 1.1
    lazy refcounts: false

image: rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf
file format: raw
virtual size: 5.0G (5368709120 bytes)
disk size: unavailable
cluster_size: 4194304

Actual results:
In step 5, active commit with rbd based disk does NOT work
In step 6, inactive commit works well

Expected results:
Support active commit with rbd based disk

Additional info:
2015-02-06 03:09:43.200+0000: 19024: debug : qemuMonitorIOProcess:399 : QEMU_MONITOR_IO_PROCESS: mon=0x7fba70010070 buf={"id": "libvirt-100", "error": {"class": "GenericError", "desc": "Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}^M
 len=152
2015-02-06 03:09:43.200+0000: 19024: debug : qemuMonitorJSONIOProcessLine:183 : Line [{"id": "libvirt-100", "error": {"class": "GenericError", "desc": "Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}]
2015-02-06 03:09:43.200+0000: 19024: debug : virJSONValueFromString:1136 : string={"id": "libvirt-100", "error": {"class": "GenericError", "desc": "Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}
2015-02-06 03:09:43.200+0000: 19024: debug : qemuMonitorJSONIOProcessLine:203 : QEMU_MONITOR_RECV_REPLY: mon=0x7fba70010070 reply={"id": "libvirt-100", "error": {"class": "GenericError", "desc": "Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}
2015-02-06 03:09:43.200+0000: 19024: debug : qemuMonitorJSONIOProcess:252 : Total used 152 bytes out of 152 available in buffer
2015-02-06 03:09:43.201+0000: 19025: debug : virJSONValueToString:1303 : result={"id":"libvirt-100","error":{"class":"GenericError","desc":"Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}
2015-02-06 03:09:43.201+0000: 19024: debug : virEventPollCleanupHandles:576 : Cleanup 13
2015-02-06 03:09:43.201+0000: 19025: debug : qemuMonitorJSONCheckError:370 : unable to execute QEMU command {"execute":"block-commit","arguments":{"device":"drive-virtio-disk0","speed":0,"top":"/tmp/vm1.s3","base":"rbd:libvirt-pool/rbd1.img:auth_supported=none:conf=/etc/ceph.conf"},"id":"libvirt-100"}: {"id":"libvirt-100","error":{"class":"GenericError","desc":"Block format 'rbd' used by device '' does not support feature 'reopening of file'"}}

Comment 2 rhbug 2016-03-14 16:13:49 UTC
Is there currently any progress on it? Could you provide us an update, when it could possibly be fixed?

Comment 3 Ademar Reis 2016-12-21 19:06:48 UTC
(In reply to rhbug from comment #2)
> Is there currently any progress on it? Could you provide us an update, when
> it could possibly be fixed?

Thanks for taking the time to enter a bug report with us. We use reports like yours to keep improving the quality of our products and releases. That said, we're not able to guarantee the timeliness or suitability of a resolution for issues entered here because this is not a mechanism for requesting support.

If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization that will result in a timely resolution.

For information on how to contact the Red Hat production support team, please visit: https://www.redhat.com/support/process/production/#howto

Comment 4 Jeff Cody 2017-04-05 19:02:48 UTC
Patches submitted upstream, for QEMU 2.10:
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg00729.html

Comment 5 Jeff Cody 2017-04-20 15:56:29 UTC
Patches applied to my block branch:
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03374.html

Comment 6 Miroslav Rezanina 2017-05-05 12:37:28 UTC
Fix included in qemu-kvm-rhev-2.9.0-3.el7

Comment 8 Suqin Huang 2017-05-25 09:00:42 UTC
Hi Jeff,

Package: qemu-kvm-rhev-2.9.0-6.el7.x86_64

Could you check if something wrong in my syntax?


{ "execute": "block-commit", "arguments": { "device": "drive_image1", "base": "rbd:kvm-test/rhel74-64-virtio.raw:mon_host=10.73.199.64", "top": "/root/sn3", "speed": 1000000000 } }


{"error": {"class": "GenericError", "desc": "Base 'rbd:kvm-test/rhel74-64-virtio.raw:mon_host=10.73.199.64' not found"}}

Comment 9 Jeff Cody 2017-06-08 12:03:34 UTC
Hi Suqin,

What does your block chain and qemu command look like?  The error indicates that we are not able to find the image with that filename.  Can you show the output of the QMP command { "execute": "query-block" }?

Comment 10 Suqin Huang 2017-06-12 06:17:48 UTC
{ "execute": "query-block" }

{"return": [{"io-status": "ok", "device": "drive_image1", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"backing-image": {"backing-image": {"backing-image": {"virtual-size": 42212254720, "filename": "json:{\"driver\": \"raw\", \"file\": {\"pool\": \"kvm-test\", \"image\": \"rhel74-64-virtio.raw\", \"driver\": \"rbd\", \"=keyvalue-pairs\": \"[\\\"mon_host\\\", \\\"10.73.199.64\\\"]\"}}", "cluster-size": 4194304, "format": "raw", "dirty-flag": false}, "backing-filename-format": "raw", "virtual-size": 42212254720, "filename": "/root/sn1", "cluster-size": 65536, "format": "qcow2", "actual-size": 9187328, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "json:{\"driver\": \"raw\", \"file\": {\"pool\": \"kvm-test\", \"image\": \"rhel74-64-virtio.raw\", \"driver\": \"rbd\", \"=keyvalue-pairs\": \"[\\\"mon_host\\\", \\\"10.73.199.64\\\"]\"}}", "backing-filename": "json:{\"driver\": \"raw\", \"file\": {\"pool\": \"kvm-test\", \"image\": \"rhel74-64-virtio.raw\", \"driver\": \"rbd\", \"=keyvalue-pairs\": \"[\\\"mon_host\\\", \\\"10.73.199.64\\\"]\"}}", "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 42212254720, "filename": "/root/sn2", "cluster-size": 65536, "format": "qcow2", "actual-size": 7680000, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/root/sn1", "backing-filename": "/root/sn1", "dirty-flag": false}, "backing-filename-format": "qcow2", "virtual-size": 42212254720, "filename": "/root/sn3", "cluster-size": 65536, "format": "qcow2", "actual-size": 58859520, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "full-backing-filename": "/root/sn2", "backing-filename": "/root/sn2", "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "#block2249", "backing_file_depth": 3, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "backing_file": "/root/sn2", "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/root/sn3", "encryption_key_missing": false}, "type": "unknown"}]}

Comment 11 Suqin Huang 2017-06-12 06:23:28 UTC
Steps:

1. { "execute": "qmp_capabilities" }

2. Create sn1 ~ sn3

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn1", "format": "qcow2" } }

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn2", "format": "qcow2" } }

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn3", "format": "qcow2" } }


3. do block commit

{ "execute": "block-commit", "arguments": { "device": "drive_image1",
"base": "", "top": "/root/sn3, "speed": 1000000000 } }

cmd:

/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga std  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/src,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-src0,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=raw,file=rbd:kvm-test/rhel74-64-virtio.raw:mon_host=10.73.199.64 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \
    -device virtio-net-pci,mac=9a:04:0e:0f:10:59,id=idKK4Mdq,vectors=4,netdev=idgjGCvt,bus=pci.0,addr=05  \
    -netdev tap,id=idgjGCvt,vhost=on,script=/etc/qemu-ifup \
    -m 2048 \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu Opteron_G3 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio

Comment 12 Jeff Cody 2017-06-13 16:55:16 UTC
The backing file created from the snapshots is in json format, which is likely the reason it is not being found from the the filename URI.

In this particular test case, you could omit the base name, as it will commit to the deepest layer by default.

Comment 14 Suqin Huang 2017-06-14 03:40:14 UTC
Reproduce with 
qemu-kvm-rhev-2.9.0-2.el7.x86_64

{"error": {"class": "GenericError", "desc": "Block format 'rbd' used by node '#block089' does not support reopening files"}}


Fixed on 

qemu-kvm-rhev-2.9.0-9.el7.x86_64

{ "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn3", "speed": 1000000000 } }
{"return": {}}


(qemu) info block-jobs 
Type commit, device drive_image1: Completed 19595264 of 19595264 bytes, speed limit 1000000000 bytes/s

Steps:

{ "execute": "qmp_capabilities" }

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn1", "format": "qcow2" } }
{"return": {}}
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn2", "format": "qcow2" } }
{"return": {}}
{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn3", "format": "qcow2" } }
{"return": {}}

{ "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn3", "speed": 1000000000 } }
{"return": {}}

{ "execute": "block-job-complete", "arguments": { "device": "drive_image1" } }
{"return": {}}

Comment 15 Suqin Huang 2017-06-14 03:42:56 UTC
According to comment14, update bug status to VERIFIED

Comment 17 errata-xmlrpc 2017-08-01 23:27:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 18 errata-xmlrpc 2017-08-02 01:04:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 19 errata-xmlrpc 2017-08-02 01:56:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 20 errata-xmlrpc 2017-08-02 02:37:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 21 errata-xmlrpc 2017-08-02 03:02:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 22 errata-xmlrpc 2017-08-02 03:22:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392