Bug 1465320
| Summary: | active block commit failed with nbd backend | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Suqin Huang <shuang> |
| Component: | qemu-kvm | Assignee: | Eric Blake <eblake> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | aihua liang <aliang> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | medium | ||
| Version: | --- | CC: | aliang, chayang, coli, eblake, juzhang, knoel, qzhang, rbalakri, virt-maint, xutian |
| Target Milestone: | rc | ||
| Target Release: | 8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-07 10:19:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1473046 | ||
Live commit has always required the ability to convert a backing file from read-only to read-write (if the backing file is not already read-write). Specifying NBD as a backing file when starting qemu defaults to using NBD read-only, and there is no way to convert NBD from read-only to read-write at run-time. But your use case is creating NBD as a read-write file (the initial active image), then adding snapshots on top: > -drive > id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2, > file=nbd://10.73.196.127:9000 \ > -device > virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \ Where it gets interesting is that you are showing a difference between live commit (merge the active image to the base NBD file, then make the base NBD file active again): > { "execute": "block-commit", "arguments": { "device": "drive_image1", "top": > "/root/sn3", "speed": 1000000000 } } > {"error": {"class": "GenericError", "desc": "Block format 'nbd' used by node > '#block084' does not support reopening files"}} > from intermediate commit (shrinking the backing chain, but keeping /root/sn3 as the live image): > > pass if run > { "execute": "block-commit", "arguments": { "device": "drive_image1", "top": > "/root/sn2", "speed": 1000000000 } } > So you may have indeed discovered an oddity in the qemu code for live-commit (both versions of commit SHOULD be able to succeed, if the NBD image is indeed read-write). I'll have to investigate further. reproduce with qemu-kvm-rhev-2.9.0-2.el7.x86_64, not regression For my own notes, pared down reproducer:
window 1:
qemu-img create -f qcow2 bar 1m
qemu-nbd -t -f raw bar
window 2:
./x86_64-softmmu/qemu-system-x86_64 -nographic -nodefaults -qmp stdio \
-drive id=drive-image1,if=none,format=qcow2,file=nbd://localhost:10809 \
-device virtio-blk-pci,id=image1,drive=drive-image1
{'execute':'qmp_capabilities'}
{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar1','format':'qcow2'}}
{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar2','format':'qcow2'}}
{'execute':'blockdev-snapshot-sync','arguments':{'device':'drive-image1', 'snapshot-file':'bar3','format':'qcow2'}}
intermediate works:
{'execute':'block-commit','arguments':{'device':'drive-image1','top':'bar2'}}
active fails:
{'execute':'block-commit','arguments':{'device':'drive-image1','top':'bar3'}}
This may need a resolution similar to bug 1529666 where LUKS reports the same error message; it may also need to wait for upstream to revamp block job support. The following case also hit this error:
1. Prepare a NBD backend image
# qemu-img info nbd:10.0.1.8:5000
image: nbd://10.0.1.8:5000
file format: qcow2
virtual size: 2.0G (2147483648 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
2. Create snapshot sn1
# qemu-img create -f qcow2 -F qcow2 -b nbd:10.0.1.8:5000 sn1.qcow2
Formatting 'sn1.qcow2', fmt=qcow2 size=2147483648 backing_file=nbd:10.0.1.8:5000 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
3. Create snapshot sn2
# qemu-img create -f qcow2 -F qcow2 -b sn1.qcow2 sn2.qcow2
Formatting 'sn2.qcow2', fmt=qcow2 size=2147483648 backing_file=sn1.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
4. Commit sn2 to base
# qemu-img commit -f qcow2 -b nbd://10.0.1.8:5000 sn2.qcow2
qemu-img: Block format 'nbd' used by node '#block474' does not support reopening files
Test on qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc, don't hit this issue any more.
Set bug's status to "Closed"->"CURRENTRELEASE"
Test Steps:
1. Expose data disk
qemu-nbd -t -p 9000 /home/data.qcow2
2. Start guest with qemu cmds:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox on \
-machine q35 \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x1 \
-m 14336 \
-smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \
-cpu 'EPYC',+kvm_pv_unhalt \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20191219-050834-hFzNWwvu,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20191219-050834-hFzNWwvu,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idPQPIm8 \
-chardev socket,id=chardev_serial0,path=/var/tmp/serial-serial0-20191219-050834-hFzNWwvu,server,nowait \
-device isa-serial,id=serial0,chardev=chardev_serial0 \
-chardev socket,id=seabioslog_id_20191219-050834-hFzNWwvu,path=/var/tmp/seabios-20191219-050834-hFzNWwvu,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20191219-050834-hFzNWwvu,iobase=0x402 \
-object iothread,id=iothread0 \
-object iothread,id=iothread1 \
-device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0,multifunction=on \
-device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
-device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x2.0x1 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
-blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x2.0x2 \
-blockdev node-name=file_data1,driver=nbd,server.host=10.73.196.71,server.port=9000,server.type=inet,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_data1,driver=raw,cache.direct=on,cache.no-flush=off,file=file_data1 \
-device virtio-blk-pci,id=data1,drive=drive_data1,write-cache=on,bus=pcie.0-root-port-4,iothread=iothread1 \
-device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x2.0x3 \
-device virtio-net-pci,mac=9a:aa:82:61:c1:00,id=idmt8SOs,netdev=idsyxRmM,bus=pcie.0-root-port-5,addr=0x0 \
-netdev tap,id=idsyxRmM,vhost=on \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :1 \
-rtc base=utc,clock=host,driftfix=slew \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,slot=6,chassis=6,addr=0x2.0x4 \
-monitor stdio \
-qmp tcp:0:3000,server,nowait \
3. Create snapshot target in advance.
{'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn1','size':2147483648},'job-id':'job1'}}
{'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn1','filename':'/root/sn1'}}
{'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn1','size':2147483648,'backing-file':'json:{"driver":"raw","file":{"driver":"nbd","server.host":"10.73.196.71","server.port":"9000","server.type":"inet"}}','backing-fmt':'raw'},'job-id':'job2'}}
{'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn1','file':'drive_sn1','backing':null}}
4. Do snapshot
{'execute':'blockdev-snapshot','arguments':{'node':'drive_data1','overlay':'sn1'}}
5. In guest, dd a new file
(guest)# dd if=/dev/urandom of=test bs=1M count=1000
6. Do block commit
{'execute': 'block-commit', 'arguments': { 'device': 'sn1','job-id':'j1'}}
{"timestamp": {"seconds": 1578389387, "microseconds": 486817}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1578389387, "microseconds": 486891}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}
{"timestamp": {"seconds": 1578389391, "microseconds": 447252}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}}
{"timestamp": {"seconds": 1578389391, "microseconds": 447372}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 1150943232, "offset": 1150943232, "speed": 0, "type": "commit"}}
7. Complete the block job
{ "execute": "block-job-complete", "arguments": { "device": "j1" } }
{"return": {}}
{"timestamp": {"seconds": 1578389405, "microseconds": 678954}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679022}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679154}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 1150943232, "offset": 1150943232, "speed": 0, "type": "commit"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679230}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
{"timestamp": {"seconds": 1578389405, "microseconds": 679282}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}
|
Description of problem: "Block format 'nbd' used by node '#block084' does not support reopening files" error when do active commit Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.9.0-14.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. { "execute": "qmp_capabilities" } {"return": {}} { "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn1", "format": "qcow2" } } {"return": {}} { "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn2", "format": "qcow2" } } {"return": {}} { "execute": "blockdev-snapshot-sync", "arguments": { "device": "drive_image1", "snapshot-file": "/root/sn3", "format": "qcow2" } } {"return": {}} { "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn3", "speed": 1000000000 } } {"error": {"class": "GenericError", "desc": "Block format 'nbd' used by node '#block084' does not support reopening files"}} pass if run { "execute": "block-commit", "arguments": { "device": "drive_image1", "top": "/root/sn2", "speed": 1000000000 } } cmd: /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga std \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -device pvpanic,ioport=0x505,id=id4FDckd \ -chardev socket,id=serial_id_serial0,path=/tmp/serial,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20160129-133009-OkYQqhNt,path=/tmp/seabios-20160129-133009-OkYQqhNt,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20160129-133009-OkYQqhNt,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \ -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=nbd://10.73.196.127:9000 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=03 \ -device virtio-net-pci,mac=9a:0d:0e:0f:10:11,id=idKK4Mdq,vectors=4,netdev=idgjGCvt,bus=pci.0,addr=05 \ -netdev tap,id=idgjGCvt,vhost=on,script=/etc/qemu-ifup \ -m 2048 \ -smp 2,maxcpus=2,cores=2,threads=1,sockets=1 \ -cpu Westmere \ -vnc :2 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -no-kvm \ -monitor stdio 2. 3. Actual results: Expected results: Additional info: