Bug 1851847
| Summary: | Snapshot file not emptied after commit operation when using NFSv3 | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Xueqiang Wei <xuwei> |
| Component: | qemu-kvm | Assignee: | Kevin Wolf <kwolf> |
| qemu-kvm sub component: | Storage | QA Contact: | Tingting Mao <timao> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | chayang, coli, jinzhao, juzhang, kwolf, qzhang, timao, virt-maint, xuwei, zhenyzha |
| Version: | 9.0 | Keywords: | Reopened, Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-13 09:41:06 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Also hit it on rhel8.4. Versions: kernel-4.18.0-289.el8.x86_64 qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. Still hit this issue in qemu-kvm-6.2.0-1.el9. And a simpler reproducer is:
Nfs mount info:
# nfsstat -m
/mnt from localhost:/home/nfs_share
Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp6,timeo=600,retrans=2,sec=sys,mountaddr=::1,mountvers=3,mountport=20048,mountproto=udp6,local_lock=none,addr=::1
Steps:
# qemu-img create -f qcow2 base.qcow2 5G
# qemu-img create -f qcow2 --object secret,id=sec0,data=snapshot -b base.qcow2 sn.qcow2 -F qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0
# qemu-io --object secret,id=sec0,data=snapshot -c 'write -P1 0 1G' 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}'
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 1076170752,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"encrypt": {
"ivgen-alg": "plain64",
"hash-alg": "sha256",
"cipher-alg": "aes-256",
"uuid": "855b77e4-332c-4dd9-b979-9abb656d0fab",
"format": "luks",
"cipher-mode": "xts",
"slots": [
{
"active": true,
"iters": 1579180,
"key-offset": 4096,
"stripes": 4000
},
{
"active": false,
"key-offset": 262144
},
{
"active": false,
"key-offset": 520192
},
{
"active": false,
"key-offset": 778240
},
{
"active": false,
"key-offset": 1036288
},
{
"active": false,
"key-offset": 1294336
},
{
"active": false,
"key-offset": 1552384
},
{
"active": false,
"key-offset": 1810432
}
],
"payload-offset": 2068480,
"master-key-iters": 395031
},
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"encrypted": true,
"backing-filename": "base.qcow2",
"dirty-flag": false
}
# qemu-img commit -p --object secret,id=sec0,data=snapshot 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}'
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 1076170752, ------------------------------------> Not emptied.
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"encrypt": {
"ivgen-alg": "plain64",
"hash-alg": "sha256",
"cipher-alg": "aes-256",
"uuid": "855b77e4-332c-4dd9-b979-9abb656d0fab",
"format": "luks",
"cipher-mode": "xts",
"slots": [
{
"active": true,
"iters": 1579180,
"key-offset": 4096,
"stripes": 4000
},
{
"active": false,
"key-offset": 262144
},
{
"active": false,
"key-offset": 520192
},
{
"active": false,
"key-offset": 778240
},
{
"active": false,
"key-offset": 1036288
},
{
"active": false,
"key-offset": 1294336
},
{
"active": false,
"key-offset": 1552384
},
{
"active": false,
"key-offset": 1810432
}
],
"payload-offset": 2068480,
"master-key-iters": 395031
},
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"encrypted": true,
"backing-filename": "base.qcow2",
"dirty-flag": false
}
While the snapshot file is a general qcow2 file not encrypted. There is no the issues.
# qemu-img create -f qcow2 base.qcow2 5G
# qemu-img create -f qcow2 -F qcow2 -b base.qcow2 sn.qcow2
# qemu-io -c 'write -P1 0 1G' -f qcow2 sn.qcow2
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 1074073600,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"backing-filename": "base.qcow2",
"dirty-flag": false
}
# qemu-img commit -f qcow2 sn.qcow2
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 262144, --------------------------------------> Emptied
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"backing-filename": "base.qcow2",
"dirty-flag": false
}
Plus, tried the steps in local file system, there is no the issue.
Tested with:
qemu-kvm-6.2.0-1.el9
Local XFS file system
Steps:
# qemu-img create -f qcow2 base.qcow2 5G
# qemu-img create -f qcow2 --object secret,id=sec0,data=snapshot -b base.qcow2 sn.qcow2 -F qcow2 -o encrypt.format=luks,encrypt.key-secret=sec0
# qemu-io --object secret,id=sec0,data=snapshot -c 'write -P1 0 1G' 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}'
# qemu-img info --output=json sn.qcow2
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 1076891648,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"encrypt": {
"ivgen-alg": "plain64",
"hash-alg": "sha256",
"cipher-alg": "aes-256",
"uuid": "e89ce157-3b8e-48ab-9974-a8d65a19db8e",
"format": "luks",
"cipher-mode": "xts",
"slots": [
{
"active": true,
"iters": 1580124,
"key-offset": 4096,
"stripes": 4000
},
{
"active": false,
"key-offset": 262144
},
{
"active": false,
"key-offset": 520192
},
{
"active": false,
"key-offset": 778240
},
{
"active": false,
"key-offset": 1036288
},
{
"active": false,
"key-offset": 1294336
},
{
"active": false,
"key-offset": 1552384
},
{
"active": false,
"key-offset": 1810432
}
],
"payload-offset": 2068480,
"master-key-iters": 394172
},
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"encrypted": true,
"backing-filename": "base.qcow2",
"dirty-flag": false
}
# qemu-img commit -p --object secret,id=sec0,data=snapshot 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}'
# qemu-img info --output=json sn.qcow2
{
"backing-filename-format": "qcow2",
"virtual-size": 5368709120,
"filename": "sn.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 4194304, -------------------------------------------> Emptied.
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"encrypt": {
"ivgen-alg": "plain64",
"hash-alg": "sha256",
"cipher-alg": "aes-256",
"uuid": "e89ce157-3b8e-48ab-9974-a8d65a19db8e",
"format": "luks",
"cipher-mode": "xts",
"slots": [
{
"active": true,
"iters": 1580124,
"key-offset": 4096,
"stripes": 4000
},
{
"active": false,
"key-offset": 262144
},
{
"active": false,
"key-offset": 520192
},
{
"active": false,
"key-offset": 778240
},
{
"active": false,
"key-offset": 1036288
},
{
"active": false,
"key-offset": 1294336
},
{
"active": false,
"key-offset": 1552384
},
{
"active": false,
"key-offset": 1810432
}
],
"payload-offset": 2068480,
"master-key-iters": 394172
},
"corrupt": false,
"extended-l2": false
}
},
"full-backing-filename": "base.qcow2",
"encrypted": true,
"backing-filename": "base.qcow2",
"dirty-flag": false
}
Unfortunately we never had a developer look at this one, it seems. This behaviour is expected.
There are two different ways the qcow2 driver uses for emptying the overlay: In the common simple case, it can just create completely new metadata and truncate the image file. This is very fast and works on NFSv3, too. The conditions for this to be safe are:
/* The following function only works for qcow2 v3 images (it
* requires the dirty flag) and only as long as there are no
* features that reserve extra clusters (such as snapshots,
* LUKS header, or persistent bitmaps), because it completely
* empties the image. Furthermore, the L1 table and three
* additional clusters (image header, refcount table, one
* refcount block) have to fit inside one refcount block. It
* only resets the image file, i.e. does not work with an
* external data file. */
When this isn't possible - in your case because of the LUKS encryption - it falls back to discarding all individual data clusters to empty the image. In every case the clusters are marked as unallocated in the qcow2 metadata so that they can be reused by later allocations. If the file system supports hole punching, the space is freed in the image file, too. However, NFSv3 does not support hole punching (this was introduced in NFS 4.2).
So in order to verify this, I would ask you to check "qemu-img map --output=json" to make sure that the image is empty on the qcow2 level. If you want, you can additionally try writing new data to the overlay. The image file size should not change because it will reuse previously freed clusters (up to the old image file size, of course - when adding more data, it will have to allocate new space).
Thanks Kevin's info, tried with the steps in comment9, and keep on writing data to the sn file, the actual-size still remains ~1G, but not increased to 2G. So close this bug accordingly. Check the image info after commiting: # qemu-img commit -p --object secret,id=sec0,data=snapshot 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}' (100.00/100%) Image committed. # qemu-img info --output=json sn.qcow2 { "backing-filename-format": "qcow2", "virtual-size": 5368709120, "filename": "sn.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 1076170752, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "encrypt": { "ivgen-alg": "plain64", "hash-alg": "sha256", "cipher-alg": "aes-256", "uuid": "238e0357-fe7b-468a-9d4f-5c4e120f1521", "format": "luks", "cipher-mode": "xts", "slots": [ { "active": true, "iters": 1936996, "key-offset": 4096, "stripes": 4000 }, { "active": false, "key-offset": 262144 }, { "active": false, "key-offset": 520192 }, { "active": false, "key-offset": 778240 }, { "active": false, "key-offset": 1036288 }, { "active": false, "key-offset": 1294336 }, { "active": false, "key-offset": 1552384 }, { "active": false, "key-offset": 1810432 } ], "payload-offset": 2068480, "master-key-iters": 484790 }, "corrupt": false, "extended-l2": false } }, "full-backing-filename": "base.qcow2", "encrypted": true, "backing-filename": "base.qcow2", "dirty-flag": false } Keep on writing another 1G data to the sn file and check the info of the file again: # qemu-io --object secret,id=sec0,data=snapshot -c 'write -P1 1G 1G' 'json:{"file": {"driver": "file", "filename": "sn.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sec0"}' # qemu-img info --output=json sn.qcow2 { "backing-filename-format": "qcow2", "virtual-size": 5368709120, "filename": "sn.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 1078267904, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "encrypt": { "ivgen-alg": "plain64", "hash-alg": "sha256", "cipher-alg": "aes-256", "uuid": "238e0357-fe7b-468a-9d4f-5c4e120f1521", "format": "luks", "cipher-mode": "xts", "slots": [ { "active": true, "iters": 1936996, "key-offset": 4096, "stripes": 4000 }, { "active": false, "key-offset": 262144 }, { "active": false, "key-offset": 520192 }, { "active": false, "key-offset": 778240 }, { "active": false, "key-offset": 1036288 }, { "active": false, "key-offset": 1294336 }, { "active": false, "key-offset": 1552384 }, { "active": false, "key-offset": 1810432 } ], "payload-offset": 2068480, "master-key-iters": 484790 }, "corrupt": false, "extended-l2": false } }, "full-backing-filename": "base.qcow2", "encrypted": true, "backing-filename": "base.qcow2", "dirty-flag": false } |
Description of problem: The snapshot file was not emptied after commit operation when the backend is nfs v3. Version-Release number of selected component (if applicable): kernel-4.18.0-209.el8.x86_64 qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420 How reproducible: 5/5 Steps to Reproduce: 1. mount nfs with vers=3 # mount -t nfs -o soft,vers=3 10.73.114.12:/home/nfs_server/ /home/kvm_autotest_root/images/ 2. create snapshot image # qemu-img create --object secret,id=image1_encrypt0,data=redhat --object secret,id=sn1_encrypt0,data=redhat -f qcow2 -b 'json:{"file": {"driver": "file", "filename": "/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "image1_encrypt0"}' -o encrypt.format=luks,encrypt.key-secret=sn1_encrypt0 /home/kvm_autotest_root/images/sn1.qcow2 20G 3. boot sn1 image and create a file in guest /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 14336 \ -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2 \ -cpu 'Opteron_G5',+kvm_pv_unhalt \ -chardev socket,server,nowait,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_vm8cmutk/monitor-qmpmonitor1-20200629-034105-fGR4Xfd7 \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,server,nowait,id=qmp_id_catch_monitor,path=/var/tmp/avocado_vm8cmutk/monitor-catch_monitor-20200629-034105-fGR4Xfd7 \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=id0iLpjt \ -chardev socket,server,nowait,id=chardev_serial0,path=/var/tmp/avocado_vm8cmutk/serial-serial0-20200629-034105-fGR4Xfd7 \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20200629-034105-fGR4Xfd7,path=/var/tmp/avocado_vm8cmutk/seabios-20200629-034105-fGR4Xfd7,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20200629-034105-fGR4Xfd7,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object secret,id=image1_encrypt0,data=redhat \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/sn1.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,encrypt.format=luks,encrypt.key-secret=image1_encrypt0,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:f7:55:c0:2f:d6,id=idkGOv36,netdev=idun79zE,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idun79zE,vhost=on \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -monitor stdio \ # dd if=/dev/urandom of=/var/tmp/test.img bs=1M count=1024 # sync # md5sum /home/test.img > sn1_log # poweroff 4. check sn1 info # qemu-img info sn1.qcow2 --output=json { "virtual-size": 21474836480, "filename": "sn1.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 1171275776, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "encrypt": { "ivgen-alg": "plain64", "hash-alg": "sha256", "cipher-alg": "aes-256", "uuid": "defacaeb-f168-4998-a25d-8dc8858831d1", "format": "luks", "cipher-mode": "xts", "slots": [ { "active": true, "iters": 896524, "key-offset": 4096, "stripes": 4000 }, { "active": false, "key-offset": 262144 }, { "active": false, "key-offset": 520192 }, { "active": false, "key-offset": 778240 }, { "active": false, "key-offset": 1036288 }, { "active": false, "key-offset": 1294336 }, { "active": false, "key-offset": 1552384 }, { "active": false, "key-offset": 1810432 } ], "payload-offset": 2068480, "master-key-iters": 223825 }, "corrupt": false } }, "full-backing-filename": "json:{\"file\": {\"driver\": \"file\", \"filename\": \"/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2\"}, \"driver\": \"qcow2\", \"encrypt.key-secret\": \"image1_encrypt0\"}", "encrypted": true, "backing-filename": "json:{\"file\": {\"driver\": \"file\", \"filename\": \"/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2\"}, \"driver\": \"qcow2\", \"encrypt.key-secret\": \"image1_encrypt0\"}", "dirty-flag": false } 5. commit snapshot to backingfile # qemu-img commit --object secret,id=image1_encrypt0,data=redhat --object secret,id=sn1_encrypt0,data=redhat 'json:{"file": {"driver": "file", "filename": "/home/kvm_autotest_root/images/sn1.qcow2"}, "driver": "qcow2", "encrypt.key-secret": "sn1_encrypt0"}' # sync 6. Verify the snapshot file whether emptied after committing # qemu-img info sn1.qcow2 --output=json { "virtual-size": 21474836480, "filename": "sn1.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 1168457728, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "encrypt": { "ivgen-alg": "plain64", "hash-alg": "sha256", "cipher-alg": "aes-256", "uuid": "defacaeb-f168-4998-a25d-8dc8858831d1", "format": "luks", "cipher-mode": "xts", "slots": [ { "active": true, "iters": 896524, "key-offset": 4096, "stripes": 4000 }, { "active": false, "key-offset": 262144 }, { "active": false, "key-offset": 520192 }, { "active": false, "key-offset": 778240 }, { "active": false, "key-offset": 1036288 }, { "active": false, "key-offset": 1294336 }, { "active": false, "key-offset": 1552384 }, { "active": false, "key-offset": 1810432 } ], "payload-offset": 2068480, "master-key-iters": 223825 }, "corrupt": false } }, "full-backing-filename": "json:{\"file\": {\"driver\": \"file\", \"filename\": \"/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2\"}, \"driver\": \"qcow2\", \"encrypt.key-secret\": \"image1_encrypt0\"}", "encrypted": true, "backing-filename": "json:{\"file\": {\"driver\": \"file\", \"filename\": \"/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2\"}, \"driver\": \"qcow2\", \"encrypt.key-secret\": \"image1_encrypt0\"}", "dirty-flag": false } Actual results: after step 6, sn1.qcow2 is not empty Expected results: sn1.qcow2 is empty Additional info: Also hit it on qemu-kvm-4.2.0-28.module+el8.2.1+7211+16dfe810 Tested with nfs vers=4.2, not hit this issue.