Bug 1713008
| Summary: | Live-migrating a paused VM (without shared storage) results in: "error while loading state for instance 0x0 of device '0000:00:05.0/virtio-balloon'" | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Archit Modi <amodi> | ||||||||
| Component: | qemu-kvm-rhev | Assignee: | Kevin Wolf <kwolf> | ||||||||
| Status: | CLOSED NOTABUG | QA Contact: | aihua liang <aliang> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 7.6 | CC: | coli, dasmith, eglynn, jhakimra, jinzhao, juzhang, kchamart, knoel, kwolf, lyarwood, mbooth, rbryant, sbauza, sgordon, virt-maint, vromanso | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 1713009 1717175 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2019-06-12 15:11:48 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1713009, 1717175 | ||||||||||
| Attachments: |
|
||||||||||
|
Description
Archit Modi
2019-05-22 16:53:25 UTC
I think a rough reproducer at libvirt/QEMU-level should be:
(1) Use versions (on both source and destination hosts):
- qemu-kvm-rhev-2.12.0-18.el7_6.5.x86_64
- libvirt-4.5.0-10.el7_6.9.x86_64
(2) Create a VM with its disk located *not* on a shared storage.
(3) Pause the VM.
(4) Live-migrate the VM (including its storage) to another host, e.g.
$> virsh migrate --verbose \
--copy-storage-inc \
--p2p \
--live \
vm1 qemu+ssh://root@dest/system
(In reply to Kashyap Chamarthy from comment #8) > I think a rough reproducer at libvirt/QEMU-level should be: > > (1) Use versions (on both source and destination hosts): > > - qemu-kvm-rhev-2.12.0-18.el7_6.5.x86_64 > - libvirt-4.5.0-10.el7_6.9.x86_64 > > (2) Create a VM with its disk located *not* on a shared storage. > > (3) Pause the VM. > > (4) Live-migrate the VM (including its storage) to another host, e.g. > > $> virsh migrate --verbose \ > --copy-storage-inc \ > --p2p \ > --live \ > vm1 qemu+ssh://root@dest/system When you say "should be": could you or someone reproduce this yet? (In reply to Ademar Reis from comment #12) > (In reply to Kashyap Chamarthy from comment #8) > > I think a rough reproducer at libvirt/QEMU-level should be: > > > > (1) Use versions (on both source and destination hosts): > > > > - qemu-kvm-rhev-2.12.0-18.el7_6.5.x86_64 > > - libvirt-4.5.0-10.el7_6.9.x86_64 > > > > (2) Create a VM with its disk located *not* on a shared storage. > > > > (3) Pause the VM. > > > > (4) Live-migrate the VM (including its storage) to another host, e.g. > > > > $> virsh migrate --verbose \ > > --copy-storage-inc \ > > --p2p \ > > --live \ > > vm1 qemu+ssh://root@dest/system > > When you say "should be": could you or someone reproduce this yet? I said "should be" because I deduced the reproducer based on the OpenStack CI test. So no, I haven't run the reproducer myself, I'm afraid. I would expect the bug reporter to provide that information. Created attachment 1577485 [details]
Source QEMU (command-line) log
Created attachment 1577486 [details]
Destination QEMU (command-line) log
Can't reproduce it on qemu layer with qemu-kvm-rhev-2.12.0-18.el7_6.5.x86_64.
Test steps:
1.Start dst guest with qemu cmds:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-215744-4gqhxTV6,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-215744-4gqhxTV6,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idKnSrhI \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-215744-4gqhxTV6,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20190602-215744-4gqhxTV6,path=/var/tmp/seabios-20190602-215744-4gqhxTV6,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20190602-215744-4gqhxTV6,iobase=0x402 \
-device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
-drive id=drive_image1,if=none,snapshot=on,aio=threads,cache=none,format=qcow2,file=/home/mirror.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \
-device virtio-net-pci,mac=9a:fb:fc:fd:fe:ff,id=idrI84Jx,vectors=4,netdev=idZnFQVB,bus=pci.0,addr=0x5 \
-netdev tap,id=idZnFQVB,vhost=on \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'Penryn',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm \
-monitor stdio \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-msg timestamp=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,resourcecontrol=deny \
-incoming tcp:0:5000 \
2.In dst, start NBD server and expose mirror.qcow2
{"execute":"nbd-server-start","arguments":{"addr":{"type":"inet","data":{"host":"10.73.224.214","port":"3333"}}}}
{"return": {}}
{ "execute": "nbd-server-add", "arguments": { "device": "drive_image1","writable": true } }
{"return": {}}
3.In src, start guest with qemu cmds:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-215744-4gqhxTV6,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-215744-4gqhxTV6,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idKnSrhI \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-215744-4gqhxTV6,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20190602-215744-4gqhxTV6,path=/var/tmp/seabios-20190602-215744-4gqhxTV6,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20190602-215744-4gqhxTV6,iobase=0x402 \
-device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
-drive id=drive_image1,if=none,snapshot=on,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel77-64-virtio.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x4 \
-device virtio-net-pci,mac=9a:fb:fc:fd:fe:ff,id=idrI84Jx,vectors=4,netdev=idZnFQVB,bus=pci.0,addr=0x5 \
-netdev tap,id=idZnFQVB,vhost=on \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'Penryn',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm \
-monitor stdio \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-msg timestamp=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,resourcecontrol=deny \
4.In src, stop VM, then do block mirror from src to dst.
{"execute":"stop"}
{"timestamp": {"seconds": 1560325907, "microseconds": 122752}, "event": "STOP"}
{"return": {}}
{ "execute": "drive-mirror", "arguments": { "device": "drive_image1","target": "nbd://10.73.224.214:3333/drive_image1", "sync": "full","format": "raw", "mode": "existing" } }
{"timestamp": {"seconds": 1560324748, "microseconds": 62964}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "drive_image1"}}
{"timestamp": {"seconds": 1560324748, "microseconds": 63199}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "drive_image1"}}
{"return": {}}
{"timestamp": {"seconds": 1560324847, "microseconds": 563385}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "drive_image1"}}
{"timestamp": {"seconds": 1560324847, "microseconds": 563491}, "event": "BLOCK_JOB_READY", "data": {"device": "drive_image1", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "mirror"}}
5.Set migration capability in both src and dst
src:{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"pause-before-switchover","state":true}]}}
dst:{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"pause-before-switchover","state":true}]}}
6.Migrate from src to dst
{ "execute": "migrate", "arguments": { "uri": "tcp:10.73.224.214:5000"}}
7.Wait some minutes, then query migrate status
{"execute":"query-migrate"}
{"return": {"expected-downtime": 300, "status": "pre-switchover", "setup-time": 158, "total-time": 30423, "ram": {"total": 4312604672, "postcopy-requests": 0, "dirty-sync-count": 2, "page-size": 4096, "remaining": 0, "mbps": 268.64496, "transferred": 783461669, "duplicate": 863875, "dirty-pages-rate": 45064, "skipped": 0, "normal-bytes": 774172672, "normal": 189007}}}
{"execute":"block-job-cancel","arguments":{"device":"drive_image1"}}
8.Cancel block jobs
{"execute":"block-job-cancel","arguments":{"device":"drive_image1"}}
{"return": {}}
{"timestamp": {"seconds": 1560325069, "microseconds": 269156}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "drive_image1"}}
{"timestamp": {"seconds": 1560325069, "microseconds": 269211}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "drive_image1"}}
{"timestamp": {"seconds": 1560325069, "microseconds": 271907}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1560325069, "microseconds": 271960}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "drive_image1"}}
{"timestamp": {"seconds": 1560325069, "microseconds": 272000}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "drive_image1"}}
9.Migrate continue
{"execute":"migrate-continue","arguments":{"state":"pre-switchover"}}
10.Check vm status in src
(qemu)info status
VM status: paused (postmigrate)
11.Check vm status in dst
(qemu)info status
VM status: paused
12.Continue vm in dst
(qemu) cont
Actual Result:
Storage vm migration with balloon device can be executed successfully.
Thanks Aihua, yes I'm unable to reproduce this manually but frequently see libvirt/qemu crashing with CI, you can close this bug with this being more CI related (In reply to aihua liang from comment #18) > Can't reproduce it on qemu layer with qemu-kvm-rhev-2.12.0-18.el7_6.5.x86_64. Thanks for the thorough QEMU-level reproducer, Aihua. I'm going to close the bug based on your data. [...] *** Bug 1717175 has been marked as a duplicate of this bug. *** |