Bug 1465320 - active block commit failed with nbd backend
Summary: active block commit failed with nbd backend
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: ---
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 8.0
Assignee: Eric Blake
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks: 1473046
TreeView+ depends on / blocked
 
Reported: 2017-06-27 08:14 UTC by Suqin Huang
Modified: 2020-01-07 10:19 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-07 10:19:24 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Suqin Huang 2017-06-27 08:14:07 UTC
Description of problem:
"Block format 'nbd' used by node '#block084' does not support reopening files" error when do active commit

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-14.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.
{ "execute": "qmp_capabilities" }
{"return": {}}

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn1", "format": "qcow2" } }
{"return": {}}

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn2", "format": "qcow2" } }
{"return": {}}

{ "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn3", "format": "qcow2" } }
{"return": {}}

{ "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn3", "speed": 1000000000 } }
{"error": {"class": "GenericError", "desc": "Block format 'nbd' used by node '#block084' does not support reopening files"}}


pass if run 
{ "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn2", "speed": 1000000000 } }


cmd: 
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga std  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -device pvpanic,ioport=0x505,id=id4FDckd  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20160129-133009-OkYQqhNt,path=/tmp/seabios-20160129-133009-OkYQqhNt,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20160129-133009-OkYQqhNt,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=nbd://10.73.196.127:9000 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \
    -device virtio-net-pci,mac=9a:0d:0e:0f:10:11,id=idKK4Mdq,vectors=4,netdev=idgjGCvt,bus=pci.0,addr=05  \
    -netdev tap,id=idgjGCvt,vhost=on,script=/etc/qemu-ifup \
    -m 2048 \
    -smp 2,maxcpus=2,cores=2,threads=1,sockets=1  \
    -cpu Westmere \
    -vnc :2 \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -no-kvm \
    -monitor stdio 


2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Eric Blake 2017-06-28 13:15:55 UTC
Live commit has always required the ability to convert a backing file from read-only to read-write (if the backing file is not already read-write).  Specifying NBD as a backing file when starting qemu defaults to using NBD read-only, and there is no way to convert NBD from read-only to read-write at run-time.  But your use case is creating NBD as a read-write file (the initial active image), then adding snapshots on top:

>     -drive
> id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,
> file=nbd://10.73.196.127:9000 \
>     -device
> virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \

Where it gets interesting is that you are showing a difference between live commit (merge the active image to the base NBD file, then make the base NBD file active again):

> { "execute": "block-commit", "arguments": { "device": "drive_image1", "top":
> "/root/sn3", "speed": 1000000000 } }
> {"error": {"class": "GenericError", "desc": "Block format 'nbd' used by node
> '#block084' does not support reopening files"}}
> 

from intermediate commit (shrinking the backing chain, but keeping /root/sn3 as the live image):

> 
> pass if run 
> { "execute": "block-commit", "arguments": { "device": "drive_image1", "top":
> "/root/sn2", "speed": 1000000000 } }
> 

So you may have indeed discovered an oddity in the qemu code for live-commit (both versions of commit SHOULD be able to succeed, if the NBD image is indeed read-write).

I'll have to investigate further.

Comment 4 Suqin Huang 2017-06-28 23:15:47 UTC
reproduce with qemu-kvm-rhev-2.9.0-2.el7.x86_64, not regression

Comment 5 Eric Blake 2017-08-04 01:45:07 UTC
For my own notes, pared down reproducer:
window 1:
qemu-img create -f qcow2 bar 1m
qemu-nbd -t -f raw bar

window 2:
./x86_64-softmmu/qemu-system-x86_64 -nographic -nodefaults -qmp stdio \
-drive id=drive-image1,if=none,format=qcow2,file=nbd://localhost:10809 \
-device virtio-blk-pci,id=image1,drive=drive-image1

{'execute':'qmp_capabilities'}

{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar1','format':'qcow2'}}
{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar2','format':'qcow2'}}
{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar3','format':'qcow2'}}

intermediate works:
{'execute':'block-commit','arguments':{'device':'drive-image1','top':'bar2'}}

active fails:
{'execute':'block-commit','arguments':{'device':'drive-image1','top':'bar3'}}

Comment 6 Eric Blake 2018-01-17 21:00:17 UTC
This may need a resolution similar to bug 1529666 where LUKS reports the same error message; it may also need to wait for upstream to revamp block job support.

Comment 7 yilzhang 2018-02-14 03:31:31 UTC
The following case also hit this error:
1. Prepare a NBD backend image
# qemu-img info  nbd:10.0.1.8:5000
image: nbd://10.0.1.8:5000
file format: qcow2
virtual size: 2.0G (2147483648 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

2. Create snapshot sn1
# qemu-img create -f qcow2 -F qcow2 -b nbd:10.0.1.8:5000        sn1.qcow2
Formatting 'sn1.qcow2', fmt=qcow2 size=2147483648 backing_file=nbd:10.0.1.8:5000 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

3. Create snapshot sn2
# qemu-img create -f qcow2 -F qcow2      -b sn1.qcow2 sn2.qcow2
Formatting 'sn2.qcow2', fmt=qcow2 size=2147483648 backing_file=sn1.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

4. Commit sn2 to base
#  qemu-img commit -f qcow2   -b nbd://10.0.1.8:5000     sn2.qcow2
qemu-img: Block format 'nbd' used by node '#block474' does not support reopening files

Comment 9 aihua liang 2020-01-07 10:19:24 UTC
Test on qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc, don't hit this issue any more.
Set bug's status to "Closed"->"CURRENTRELEASE"

Test Steps:
 1. Expose data disk
     qemu-nbd -t -p 9000 /home/data.qcow2

 2. Start guest with qemu cmds:
     /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1 \
    -m 14336  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'EPYC',+kvm_pv_unhalt  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20191219-050834-hFzNWwvu,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20191219-050834-hFzNWwvu,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idPQPIm8 \
    -chardev socket,id=chardev_serial0,path=/var/tmp/serial-serial0-20191219-050834-hFzNWwvu,server,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20191219-050834-hFzNWwvu,path=/var/tmp/seabios-20191219-050834-hFzNWwvu,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20191219-050834-hFzNWwvu,iobase=0x402 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0,multifunction=on \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x2.0x1 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x2.0x2 \
    -blockdev node-name=file_data1,driver=nbd,server.host=10.73.196.71,server.port=9000,server.type=inet,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_data1,driver=raw,cache.direct=on,cache.no-flush=off,file=file_data1 \
    -device virtio-blk-pci,id=data1,drive=drive_data1,write-cache=on,bus=pcie.0-root-port-4,iothread=iothread1 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x2.0x3 \
    -device virtio-net-pci,mac=9a:aa:82:61:c1:00,id=idmt8SOs,netdev=idsyxRmM,bus=pcie.0-root-port-5,addr=0x0  \
    -netdev tap,id=idsyxRmM,vhost=on \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :1  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=6,chassis=6,addr=0x2.0x4 \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

  3. Create snapshot target in advance.
    {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn1','size':2147483648},'job-id':'job1'}}
    {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn1','filename':'/root/sn1'}}
    {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn1','size':2147483648,'backing-file':'json:{"driver":"raw","file":{"driver":"nbd","server.host":"10.73.196.71","server.port":"9000","server.type":"inet"}}','backing-fmt':'raw'},'job-id':'job2'}}
    {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn1','file':'drive_sn1','backing':null}}

  4. Do snapshot
    {'execute':'blockdev-snapshot','arguments':{'node':'drive_data1','overlay':'sn1'}}

  5. In guest, dd a new file
    (guest)# dd if=/dev/urandom of=test bs=1M count=1000
  
  6. Do block commit
    {'execute': 'block-commit', 'arguments': { 'device': 'sn1','job-id':'j1'}}
{"timestamp": {"seconds": 1578389387, "microseconds": 486817}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1578389387, "microseconds": 486891}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}
{"timestamp": {"seconds": 1578389391, "microseconds": 447252}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}}
{"timestamp": {"seconds": 1578389391, "microseconds": 447372}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 1150943232, "offset": 1150943232, "speed": 0, "type": "commit"}}

  7. Complete the block job
    { "execute": "block-job-complete", "arguments": { "device": "j1" } }
    {"return": {}}
{"timestamp": {"seconds": 1578389405, "microseconds": 678954}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679022}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679154}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 1150943232, "offset": 1150943232, "speed": 0, "type": "commit"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679230}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679282}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}


Note You need to log in before you can comment on or make changes to this bug.