Bug 1877724
| Summary: | Qemu core dump when migrate with migrate-mapping to an existed bitmap | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | aihua liang <aliang> |
| Component: | qemu-kvm | Assignee: | Eric Blake <eblake> |
| qemu-kvm sub component: | Incremental Live Backup | QA Contact: | Gu Nini <ngu> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | high | CC: | coli, eblake, jinzhao, juzhang, ngu, qzhang, virt-maint |
| Version: | 8.3 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-01-12 22:31:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I'm able to reproduce the crash on just one machine with this patch to iotests; I'm now working on understanding what the problem is and how to fix it.
diff --git i/tests/qemu-iotests/300 w/tests/qemu-iotests/300
index 5b75121b8496..ed578c89d032 100755
--- i/tests/qemu-iotests/300
+++ w/tests/qemu-iotests/300
@@ -589,5 +589,42 @@ class TestCrossAliasMigration(TestDirtyBitmapMigration):
self.verify_dest_error(None)
+class TestCollisionMigration(TestDirtyBitmapMigration):
+ """
+ Create a collision between incoming migration and existing bitmap.
+ """
+
+ src_node_name = 'node-a'
+ dst_node_name = 'node-b'
+ src_bmap_name = 'bmap-a'
+ dst_bmap_name = 'bmap-b'
+
+ def setUp(self) -> None:
+ TestDirtyBitmapMigration.setUp(self)
+
+ # Now pre-create the destination bitmap
+ result = self.vm_b.qmp('block-dirty-bitmap-add',
+ node='node-b', name='bmap-b')
+ self.assert_qmp(result, 'return', {})
+
+ def test_collision_on_src(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': self.src_node_name,
+ 'alias': 'node-b',
+ 'bitmaps': [{ 'name': 'bmap-a', 'alias': 'bmap-b'}]
+ }]
+
+ self.set_mapping(self.vm_a, mapping)
+
+ # Migration should succeed
+ self.migrate()
+ self.verify_dest_error(None)
+
+ def test_collision_on_dst(self) -> None:
+ # No incoming mapping needed
+ self.migrate()
+ self.verify_dest_error(None)
+
+
if __name__ == '__main__':
iotests.main(supported_protocols=['file'])
actually, migration is supposed to fail if the bitmap already exists at the destination. Still, qemu shouldn't dump core. |
Description of problem: Qemu core dump when migrate with migrate-mapping to an existed bitmap Version-Release number of selected component (if applicable): kernel version:4.18.0-233.el8.x86_64 qemu-kvm version:qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f How reproducible: 100% Steps to Reproduce: 1. Start guest with qemu cmds in src: ... -blockdev node-name=file_image1,driver=file,aio=threads,filename=/mnt/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device virtio-blk-pci,id=image1,drive=drive_image1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \ ... 2. Start guest with qemu cmds in dst: ... -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/mirror.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_img1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_img1 \ -device virtio-blk-pci,id=img1,drive=drive_img1,write-cache=on,bus=pcie-root-port-2,iothread=iothread0 \ -incoming tcp:0:5000 \ ... 3. Expose image in dst { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host": "10.66.144.75", "port": "3333" } } } } { "execute": "nbd-server-add", "arguments":{ "device": "drive_img1", "writable": true } } 4. Do mirror from src to dst {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror","server":{"type":"inet","host":"10.66.144.75","port":"3333"},"export":"drive_img1"}} {"execute": "blockdev-mirror", "arguments": { "device": "drive_image1","target": "mirror", "sync": "full", "job-id":"j1"}} {"timestamp": {"seconds": 1599730499, "microseconds": 851240}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}} {"timestamp": {"seconds": 1599730499, "microseconds": 851387}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}} {"return": {}} {"timestamp": {"seconds": 1599730576, "microseconds": 266693}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}} {"timestamp": {"seconds": 1599730576, "microseconds": 266781}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 21474902016, "offset": 21474902016, "speed": 0, "type": "mirror"}} 5. Set migration capabilities in both src and dst {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}} 6. Add bitmap0 in src image and check bitmap info { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}} {"execute":"query-block"} {"return": [{"io-status": "ok", "device": "", "locked": false, "removable": false, "inserted": {"iops_rd": 0, "detect_zeroes": "off", "image": {"virtual-size": 21474836480, "filename": "/mnt/rhel830-64-virtio-scsi.qcow2", "cluster-size": 65536, "format": "qcow2", "actual-size": 6699487232, "format-specific": {"type": "qcow2", "data": {"compat": "1.1", "compression-type": "zlib", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false}}, "dirty-flag": false}, "iops_wr": 0, "ro": false, "node-name": "drive_image1", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 655360}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "encrypted": false, "bps": 0, "bps_rd": 0, "cache": {"no-flush": false, "direct": true, "writeback": true}, "file": "/mnt/rhel830-64-virtio-scsi.qcow2", "encryption_key_missing": false}, "qdev": "/machine/peripheral/image1/virtio-backend", "dirty-bitmaps": [{"name": "bitmap0", "recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 655360}, {"recording": true, "persistent": false, "busy": false, "status": "active", "granularity": 65536, "count": 0}], "type": "unknown"}]} 7. Add bitmapa in dst image { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_img1","name":"bitmapa"}} 8. Set bitmap mapping in src {"execute":"migrate-set-parameters","arguments":{"block-bitmap-mapping":[{"node-name":"drive_image1","alias":"drive_img1","bitmaps":[{"name":"bitmap0","alias":"bitmapa"}]}]}} 9. Set pre-switchover in src {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"pause-before-switchover","state":true}]}} 10. Migrate from src to dst {"execute": "migrate","arguments":{"uri": "tcp:10.66.144.75:5000"}} {"timestamp": {"seconds": 1599730782, "microseconds": 73304}, "event": "MIGRATION", "data": {"status": "setup"}} {"return": {}} {"timestamp": {"seconds": 1599730782, "microseconds": 82751}, "event": "MIGRATION_PASS", "data": {"pass": 1}} {"timestamp": {"seconds": 1599730782, "microseconds": 82981}, "event": "MIGRATION", "data": {"status": "active"}} {"timestamp": {"seconds": 1599730782, "microseconds": 88499}, "event": "MIGRATION", "data": {"status": "failed"}} {"timestamp": {"seconds": 1599730808, "microseconds": 38836}, "event": "BLOCK_JOB_ERROR", "data": {"device": "j1", "operation": "write", "action": "report"}} {"timestamp": {"seconds": 1599730808, "microseconds": 39609}, "event": "BLOCK_JOB_ERROR", "data": {"device": "j1", "operation": "write", "action": "report"}} {"timestamp": {"seconds": 1599730808, "microseconds": 40015}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "j1"}} Ncat: Connection reset by peer. Actual results: In src, qemu coredump with info: (qemu) qemu-kvm: failed to save SaveStateEntry with id(name): 1(ram) qemu-kvm: Unable to write to socket: Connection reset by peer src.txt: line 43: 444357 Segmentation fault (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox on -machine q35 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 ... In dst: (qemu) qemu-kvm: Bitmap with the same name ('bitmapa') already exists on destination qemu-kvm: error while loading state for instance 0x0 of device 'dirty-bitmap' qemu-kvm: load of migration failed: Invalid argument And gdb info as bellow: (gdb) bt #0 0x000055e3f70f8328 in qio_channel_detach_aio_context (ioc=0x0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/io/channel.c:452 #1 0x000055e3f707c911 in bdrv_detach_aio_context (bs=0x55e3f83a6cf0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:6267 #2 0x000055e3f707c911 in bdrv_set_aio_context_ignore (bs=bs@entry=0x55e3f83a6cf0, new_context=new_context@entry=0x55e3f7d79a70, ignore=ignore@entry=0x7fff20761fb0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:6346 #3 0x000055e3f707cdc3 in bdrv_child_try_set_aio_context (bs=bs@entry=0x55e3f83a6cf0, ctx=0x55e3f7d79a70, ignore_child=ignore_child@entry=0x0, errp=errp@entry=0x0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:6450 #4 0x000055e3f707cef6 in bdrv_try_set_aio_context (errp=0x0, ctx=<optimized out>, bs=0x55e3f83a6cf0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:6459 #5 0x000055e3f707cef6 in bdrv_replace_child (child=child@entry=0x55e3f7d7f040, new_bs=new_bs@entry=0x0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:2654 #6 0x000055e3f707cf7f in bdrv_detach_child (child=0x55e3f7d7f040) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:2773 #7 0x000055e3f707d84d in bdrv_root_unref_child (child=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block.c:2784 #8 0x000055e3f7082f16 in block_job_remove_all_bdrv (job=job@entry=0x55e3f876b150) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/blockjob.c:191 #9 0x000055e3f70bea19 in mirror_exit_common (job=0x55e3f876b150) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block/mirror.c:742 #10 0x000055e3f70bf77d in mirror_abort (job=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/block/mirror.c:770 #11 0x000055e3f708448f in job_abort (job=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/job.c:670 #12 0x000055e3f708448f in job_finalize_single (job=0x55e3f876b150) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/job.c:691 #13 0x000055e3f708448f in job_finalize_single (job=0x55e3f876b150) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/job.c:681 #14 0x000055e3f7084f2a in job_completed_txn_abort (job=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging-- at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/job.c:769 #15 0x000055e3f7085185 in job_exit (opaque=0x55e3f876b150) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/job.c:885 #16 0x000055e3f715519d in aio_bh_call (bh=0x7f1e70015180) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/async.c:164 #17 0x000055e3f715519d in aio_bh_poll (ctx=ctx@entry=0x55e3f7d79a70) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/async.c:164 #18 0x000055e3f71448f2 in aio_dispatch (ctx=0x55e3f7d79a70) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/aio-posix.c:380 #19 0x000055e3f7155082 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/async.c:306 #20 0x00007f1e8992767d in g_main_dispatch (context=0x55e3f7d79ea0) at gmain.c:3176 #21 0x00007f1e8992767d in g_main_context_dispatch (context=context@entry=0x55e3f7d79ea0) at gmain.c:3829 #22 0x000055e3f7159dc8 in glib_pollfds_poll () at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/main-loop.c:217 #23 0x000055e3f7159dc8 in os_host_main_loop_wait (timeout=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/main-loop.c:240 #24 0x000055e3f7159dc8 in main_loop_wait (nonblocking=nonblocking@entry=0) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/util/main-loop.c:516 #25 0x000055e3f6eb1541 in qemu_main_loop () at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/softmmu/vl.c:1676 #26 0x000055e3f6dcf372 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /usr/src/debug/qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64/softmmu/main.c:49 Expected results: Qemu should not coredump.