Created attachment 1652046 [details] qemu.log Description of problem: Migration failed with copy-storage-all for can't find correct node_name Version-Release number of selected component (if applicable): libvirt-5.10.0-2.module+el8.2.0+5274+60f836b5.x86_64 qemu-kvm-4.2.0-5.module+el8.2.0+5389+367d9739.x86_64 How reproducible: 100% Steps to Reproduce: 1.Start a guest with local storage: ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> ... 2.Migrate the guest to another 8.2 host with copy-storage-all: # virsh migrate demo qemu+ssh://10.16.200.75/system --live --verbose --copy-storage-all root.200.75's password: error:internal error: unable to execute QEMU command 'blockdev-mirror': Cannot find device=drive-virtio-disk0 nor node_name=drive-virtio-disk0 3.guest qemu-cmd-line: # ps aux |grep demo qemu 14219 98.8 1.8 2141224 141752 ? Sl 22:37 0:07 /usr/libexec/qemu-kvm -name guest=demo,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-65-demo/master-key.aes -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-rhel7.6.0,accel=kvm,usb=off,vmport=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format -global mch.extended-tseg-mbytes=48 -cpu SandyBridge-IBRS,vme=on,ds=off,acpi=off,ss=on,ht=off,tm=off,pbe=off,dtes64=off,monitor=off,ds-cpl=off,vmx=off,smx=off,est=off,tm2=off,xtpr=off,pdcm=off,pcid=on,arat=on,stibp=on,xsaveopt=on,hypervisor=on -m 512 -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -object iothread,id=iothread1 -object iothread,id=iothread2 -object iothread,id=iothread3 -object iothread,id=iothread4 -uuid 1261efdf-cf70-489c-8138-c18b2e75497e -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=43,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x15,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x7 -device pcie-root-port,port=0x14,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,addr=0x3 -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1a -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.4,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.5,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 -netdev tap,fd=45,id=hostnet0,vhost=on,vhostfd=46 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f4:d5:8b,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on Actual results: Migration failed in step2 Expected results: Can migrate successfully with copy-storage-all,the node_name need to point to "drive=libvirt-1-format". Additional info: More additional qemu log in attachment
Fixed upstream with the following series: v5.10.0-82-gf5259ba402 commit f5259ba4027807e82ce040f63db40efe56e86925 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 18:50:46 2019 +0100 qemu: migration: Properly setup mirror for blockdev configurations With blockdev we need to refer to the nodename of the disk source image as the source argument for the blockdev-mirror operation while still keeping the old job name. With blockdev we must also persist the job in qemu. commit 3e35156bd18badf9757cad0904493e7eda530320 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 18:44:22 2019 +0100 qemu: migration: Mention disk target rather than the drive name in debug msg commit 69abc80f5e58ba2c0f30e1d7296be8524631246d Author: Peter Krempa <pkrempa> Date: Fri Dec 6 17:47:46 2019 +0100 qemu: migration: Split out setup of the migration target Separate out allocation of the virStorageSource corresponding to the target NBD export of the migration. As part of the splitout we allocate the export name explicitly as that one must not change regardless whether blockdev is used or not to provide compatibility. commit 0d0b490a32be5cb4f8a3b103784e1f10ce6c51ce Author: Peter Krempa <pkrempa> Date: Fri Dec 6 17:46:29 2019 +0100 qemu: blockjob: Allow NULL 'mirror' for block copy jobs due to migration The non-shared-storage migration tracks the storage source used explicitly in the migration data so we must allow for processing of the block job which has NULL mirror as the mirror will not be populated. commit e1910a1f3b5a1926bc130ddd16ebee0a3cf8db27 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 17:26:01 2019 +0100 qemu: migration: Simplify cleanup in qemuMigrationSrcNBDCopyCancelOne Now that the cleanup section does not exist remove the label. commit 37e801340ac6902e63e83983f5e606fbc38a23e6 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 17:22:53 2019 +0100 qemu: migration: Access job name from job struct qemuMigrationSrcNBDCopyCancelOne uses the block job data structure but generated it's own job name rather than taking it from the block job data. commit 5339f57a0bf958f5c432c6ae6b655bf234ea0971 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 17:02:33 2019 +0100 qemu: migration: Properly export backend for NBD storage migration With -blockdev we must use the nodename as the export but we must keep the name of the export as it was before to ensure compatiblity. commit 83137f5d3710ffe5a36ba9218ddc382d6675dc32 Author: Peter Krempa <pkrempa> Date: Fri Dec 6 16:54:33 2019 +0100 qemu: migration: Simplify handling of 'diskAlias' when adding NBD exports Declare the variable inside the loop with automatic clearing.
Hi Peter, Libvirtd will not pre-create images on the target host with libvirt-6.0.0-1. Is there any change in this part? Test steps: 1.The same pool in the both source and target host: # virsh pool-list Name State Autostart --------------------------------- default active no 2.Show the image of the guest: # virsh domblklist demo Target Source ----------------------------------------------------------------------------- vdb /var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2.2 3.Do storage migration: # virsh migrate demo --live --verbose qemu+ssh://10.16.200.75/system --copy-storage-all root.200.75's password: error: Cannot access storage file '/var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2.2': No such file or directory 4.Check the libvirtd log and no disk info in the qemuMigrationBakeCookie: 2020-01-17 10:24:25.682+0000: 26801: debug : qemuMigrationBakeCookie:1460 : cookielen=638 cookie=<qemu-migration> <name>demo</name> <uuid>1261efdf-cf70-489c-8138-c18b2e75497e</uuid> <hostname>jgao-test1</hostname> <hostuuid>476175ba-a0b6-c361-c59e-61f3341b621f</hostuuid> <feature name='lockstate'/> <nbd/> <allowReboot value='yes'/> <capabilities> <cap name='xbzrle' auto='no'/> <cap name='auto-converge' auto='no'/> <cap name='rdma-pin-all' auto='no'/> <cap name='postcopy-ram' auto='no'/> <cap name='compress' auto='no'/> <cap name='pause-before-switchover' auto='yes'/> <cap name='late-block-activate' auto='no'/> <cap name='multifd' auto='no'/> </capabilities> </qemu-migration>
It should work normally. What's the domain XML? Also this bug doesn't deal with the storage pre-creation at all.
(In reply to Peter Krempa from comment #4) > It should work normally. What's the domain XML? Also this bug doesn't deal > with the storage pre-creation at all. This problem happened when i tried to verify this bug,and yafu helped to debug this issue. And we found the description in man page: "--copy-storage-all indicates migration with non-shared storage with full disk copy, --copy-storage-inc indicates migration with non-shared storage with incremental copy (same base image shared between source and destination). In both cases the disk images have to exist on destina‐ tion host" But we remember that we can migrate with copying storage without "pre-create" the image in target in the latest libvirt version. So please help to confirm why it doesn't work now? The domain xml is: <domain type='kvm' id='58'> <name>demo</name> <uuid>1261efdf-cf70-489c-8138-c18b2e75497e</uuid> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>1</vcpu> <iothreads>4</iothreads> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-rhel7.6.0'>hvm</type> <loader readonly='no' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader> <nvram>/usr/share/OVMF/OVMF_VARS.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> <smm state='on'> <tseg unit='MiB'>48</tseg> </smm> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>SandyBridge-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='disable' name='ds'/> <feature policy='disable' name='acpi'/> <feature policy='require' name='ss'/> <feature policy='disable' name='ht'/> <feature policy='disable' name='tm'/> <feature policy='disable' name='pbe'/> <feature policy='disable' name='dtes64'/> <feature policy='disable' name='monitor'/> <feature policy='disable' name='ds_cpl'/> <feature policy='disable' name='vmx'/> <feature policy='disable' name='smx'/> <feature policy='disable' name='est'/> <feature policy='disable' name='tm2'/> <feature policy='disable' name='xtpr'/> <feature policy='disable' name='pdcm'/> <feature policy='require' name='pcid'/> <feature policy='require' name='arat'/> <feature policy='require' name='stibp'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='hypervisor'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2.2' index='1'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1a' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x15'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x16'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x17'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0x14'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x18'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x19'/> alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='11' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='fdc' index='0'> <alias name='fdc0'/> </controller> <interface type='network'> <mac address='52:54:00:f4:d5:8b'/> <source network='default' portid='dbb9f25f-633c-49e1-b2b8-68d7d04217bf' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich9'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0' port='3'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c450,c525</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c450,c525</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain>
At any rate, the storage pre-creation should work. If it does not, please open a separate bug as the symptoms are not relevant to this bug.
(In reply to Peter Krempa from comment #6) > At any rate, the storage pre-creation should work. If it does not, please > open a separate bug as the symptoms are not relevant to this bug. I filed a new bug Bug 1793263 to track this issue.
Verified on: libvirt-6.0.0-5.virtcov.el8.x86_64 qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64 Step: 1.Prepare a guest with local storage: # virsh domblklist demo Target Source --------------------------------------------------------------------------- vda /var/lib/libvirt/images/RHEL-8.2.0-20191219.0-x86_64-ovmf.qcow2 2.Migrate the guest with copy-storage-all to another host: # virsh migrate demo qemu+ssh://$ip/system --live --verbose --copy-storage-all root@$ip's password: Migration: [100 %] qemu log : qemuMonitorSend:996 : QEMU_MONITOR_SEND_MSG: mon=0x7f92a00615c0 msg={"execute":"blockdev-mirror","arguments":{"job-id":"drive-virtio-disk0","device":"libvirt-1-format","target":"migration-vda-format","speed":9223372036853727232,"sync":"full","auto-finalize":true,"auto-dismiss":false},"id":"libvirt-437"} QEMU_MONITOR_SEND_MSG: mon=0x7f92a00615c0 msg={"execute":"blockdev-add","arguments":{"driver":"nbd","server":{"type":"inet","host":"lenovo-sr630-06.lab.eng.pek2.redhat.com","port":"49153"},"export":"drive-virtio-disk0","node-name":"migration-vda-storage","read-only":false,"discard":"unmap"},"id":"libvirt-435"} Node_name has pointed to the specific disk name. 3.Guest works well after migration Work as expected.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017