Description of problem: Live migration of a VM with attached read only cdrom fails on: "timed out waiting for domain to be defined" Version-Release number of selected component (if applicable): CNV 4.8.0 How reproducible: 100% Steps to Reproduce: 1. Create a VM with attached readonly cdrom (default when creating a Windows VM from the UI) 2. Start the VM 3. Trigger live migration Actual results: Migration fails Expected results: Migration should succeed Additional info: $ oc get pod -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES virt-launcher-windows-vm-wth-guest-tools-1622617042-007237djn6t 2/2 Running 0 32m 10.128.2.132 virt-vk-2-288t4-worker-0-x48zb <none> <none> virt-launcher-windows-vm-wth-guest-tools-1622617042-007237vwvgw 0/2 Error 0 30m 10.129.3.38 virt-vk-2-288t4-worker-0-bqzpd <none> <none> =============== Target pod log =========================== {"component":"virt-launcher","kind":"","level":"info","msg":"Prepared migration target pod","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:167","timestamp":"2021-06-02T07:00:18.055840Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"error","msg":"At least one cgroup controller is required: No such device or address","pos":"virCgroupDetectControllers:455","subcomponent":"libvirt","thread":"35","timestamp":"2021-06-02T07:00:19.182000Z"} {"component":"virt-launcher","level":"info","msg":"GuestAgentLifecycle event state 2 with reason 1 received","pos":"client.go:458","timestamp":"2021-06-02T07:00:19.678032Z"} {"component":"virt-launcher","level":"error","msg":"failed to get domain metadata","pos":"libvirt_helper.go:141","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:19.683583Z"} {"component":"virt-launcher","level":"error","msg":"Could not fetch the Domain specification.","pos":"client.go:252","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:19.683694Z"} {"component":"virt-launcher","level":"info","msg":"Domain name event: ","pos":"client.go:377","timestamp":"2021-06-02T07:00:19.683823Z"} {"component":"virt-launcher","level":"info","msg":"2021-06-02 07:00:19.175+0000: starting up libvirt version: 7.0.0, package: 14.module+el8.4.0+10886+79296686 (Red Hat, Inc. \u003chttp://bugzilla.redhat.com/bugzilla\u003e, 2021-05-06-06:29:31, ), qemu version: 5.2.0qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207, kernel: 4.18.0-305.el8.x86_64, hostname: virt-launcher-windows-vm-wth-guest-tools-1622617042-007237vwvgw","subcomponent":"qemu","timestamp":"2021-06-02T07:00:19.922543Z"} {"component":"virt-launcher","level":"info","msg":"LC_ALL=C \\PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \\HOME=/var/lib/libvirt/qemu/domain-1-virt-readonly-disk-t \\XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-virt-readonly-disk-t/.local/share \\XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-virt-readonly-disk-t/.cache \\XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-virt-readonly-disk-t/.config \\QEMU_AUDIO_DRV=none \\/usr/libexec/qemu-kvm \\-name guest=virt-readonly-disk-test-vm-with-windows-guest-tools_windows-vm-wth-guest-tools-1622617042-0072374,debug-threads=on \\-S \\-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-virt-readonly-disk-t/master-key.aes \\-machine pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \\-cpu Haswell-noTSX-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-stimer,hv-stimer-direct,hv-reset,hv-frequencies,hv-reenlightenment,hv-tlbflush,hv-ipi \\-m 4096 \\-object memory-backend-ram,id=pc.ram,size=4294967296 \\-overcommit mem-lock=off \\-smp 2,sockets=1,dies=1,cores=1,threads=2 \\-object iothread,id=iothread1 \\-uuid 0ce8cd97-9d53-5e9f-b145-b9f0fd9d6af0 \\-smbios 'type=1,manufacturer=Red Hat,product=Container-native virtualization,version=4.8.0,uuid=0ce8cd97-9d53-5e9f-b145-b9f0fd9d6af0,sku=4.8.0,family=Red Hat' \\-no-user-config \\-nodefaults \\-chardev socket,id=charmonitor,fd=20,server,nowait \\-mon chardev=charmonitor,id=monitor,mode=control \\-rtc base=utc,driftfix=slew \\-global kvm-pit.lost_tick_policy=delay \\-no-hpet \\-no-shutdown \\-boot strict=on \\-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \\-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \\-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \\-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \\-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \\-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \\-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \\-device qemu-xhci,id=usb,bus=pci.2,addr=0x0 \\-device virtio-scsi-pci-non-transitional,id=scsi0,bus=pci.3,addr=0x0 \\-device virtio-serial-pci-non-transitional,id=virtio-serial0,bus=pci.4,addr=0x0 \\-blockdev '{\"driver\":\"host_device\",\"filename\":\"/dev/windows-vm-wth-guest-tools-1622617042-0072374\",\"aio\":\"native\",\"node-name\":\"libvirt-3-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-3-format\",\"read-only\":false,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"raw\",\"file\":\"libvirt-3-storage\"}' \\-device ide-hd,bus=ide.0,drive=libvirt-3-format,id=ua-windows-vm-wth-guest-tools-1622617042-0072374,bootindex=1,write-cache=on,werror=stop,rerror=stop \\-blockdev '{\"driver\":\"file\",\"filename\":\"/var/run/kubevirt/container-disks/disk_1.img\",\"node-name\":\"libvirt-2-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-2-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"raw\",\"file\":\"libvirt-2-storage\"}' \\-blockdev '{\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-ephemeral-disks/disk-data/windows-guest-tools/disk.qcow2\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}' \\-blockdev '{\"node-name\":\"libvirt-1-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":\"libvirt-2-format\"}' \\-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=ua-windows-guest-tools,write-cache=on,werror=stop,rerror=stop \\-netdev tap,fd=22,id=hostua-default \\-device e1000e,netdev=hostua-default,id=ua-default,mac=02:f4:4b:00:00:01,bus=pci.1,addr=0x0,romfile= \\-chardev socket,id=charserial0,fd=23,server,nowait \\-device isa-serial,chardev=charserial0,id=serial0 \\-chardev socket,id=charchannel0,fd=24,server,nowait \\-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \\-device usb-tablet,id=ua-tablet,bus=usb.0,port=1 \\-vnc vnc=unix:/var/run/kubevirt-private/862d59ef-720e-4ab9-8024-199c0f0000f5/virt-vnc \\-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \\-incoming defer \\-device virtio-balloon-pci-non-transitional,id=balloon0,bus=pci.5,addr=0x0 \\-object rng-random,id=objrng0,filename=/dev/urandom \\-device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 \\-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \\-msg timestamp=on","subcomponent":"qemu","timestamp":"2021-06-02T07:00:19.922821Z"} {"component":"virt-launcher","level":"info","msg":"DomainLifecycle event 2 with reason 1 received","pos":"client.go:401","timestamp":"2021-06-02T07:00:20.172101Z"} {"component":"virt-launcher","level":"error","msg":"failed to get domain metadata","pos":"libvirt_helper.go:141","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:20.181108Z"} {"component":"virt-launcher","level":"error","msg":"Could not fetch the Domain specification.","pos":"client.go:252","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:20.181207Z"} {"component":"virt-launcher","level":"info","msg":"Domain name event: ","pos":"client.go:377","timestamp":"2021-06-02T07:00:20.181278Z"} {"component":"virt-launcher","level":"error","msg":"failed to get domain metadata","pos":"libvirt_helper.go:141","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:27.433465Z"} {"component":"virt-launcher","level":"error","msg":"failed to list domains when detecting UUID","pos":"virt-launcher.go:224","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:27.433571Z"} {"component":"virt-launcher","level":"error","msg":"failed to get domain metadata","pos":"libvirt_helper.go:141","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:00:37.432383Z"} .... .... {"component":"virt-launcher","level":"error","msg":"failed to get domain metadata","pos":"libvirt_helper.go:141","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:05:07.433926Z"} {"component":"virt-launcher","level":"error","msg":"failed to list domains when detecting UUID","pos":"virt-launcher.go:224","reason":"virError(Code=80, Domain=20, Message='metadata not found: Requested metadata element is not present')","timestamp":"2021-06-02T07:05:07.434074Z"} panic: timed out waiting for domain to be defined goroutine 1 [running]: main.waitForDomainUUID(0x45d964b800, 0xc0003fa420, 0xc0000fa060, 0x1ca2e20, 0xc00021e900, 0xc0003fa420) /go/src/kubevirt.io/kubevirt/cmd/virt-launcher/virt-launcher.go:243 +0x4fe main.main() /go/src/kubevirt.io/kubevirt/cmd/virt-launcher/virt-launcher.go:473 +0x1018 {"component":"virt-launcher","level":"error","msg":"dirty virt-launcher shutdown","pos":"virt-launcher.go:554","reason":"exit status 2","timestamp":"2021-06-02T07:05:17.434919Z"} =============== Source pod log =========================== {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:17.971018Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"DomainLifecycle event 0 with reason 1 received","pos":"client.go:401","timestamp":"2021-06-02T07:00:17.970844Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Initiating live migration.","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"live-migration-source.go:896","timestamp":"2021-06-02T07:00:17.971096Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"No SR-IOV host-devices to detach.","pos":"hostdev.go:113","timestamp":"2021-06-02T07:00:17.974435Z"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:00:17.977358Z"} {"component":"virt-launcher","level":"info","msg":"proxy started listening","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49152-source.sock","pos":"migration-proxy.go:514","timestamp":"2021-06-02T07:00:17.979088Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"proxy started listening","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:514","timestamp":"2021-06-02T07:00:17.979205Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"proxy started listening","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-source.sock","pos":"migration-proxy.go:514","timestamp":"2021-06-02T07:00:17.979299Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"Domain name event: virt-readonly-disk-test-vm-with-windows-guest-tools_windows-vm-wth-guest-tools-1622617042-0072374","pos":"client.go:377","timestamp":"2021-06-02T07:00:17.979931Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:18.003768Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:18.087287Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:00:19.522222Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:19.526712Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:00:19.620836Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:19.628641Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:19.768310Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:00:59.535026Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:59.543646Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:00:59.645192Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:00:59.654159Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:02:19.550968Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:02:19.556823Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:02:19.684507Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:02:19.694752Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:04:19.706403Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:04:19.713996Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:04:58.504433Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:04:58.617965Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"kubevirt domain status: Running(1):Unknown(1)","pos":"client.go:262","timestamp":"2021-06-02T07:04:59.560741Z"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:04:59.568568Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Signaled vmi migration","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"server.go:113","timestamp":"2021-06-02T07:04:59.619834Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"137895 bytes copied outbound to inbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:442","timestamp":"2021-06-02T07:05:17.435665Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"9042112487 bytes copied from inbound to outbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:448","timestamp":"2021-06-02T07:05:17.435848Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"0 bytes copied outbound to inbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:442","timestamp":"2021-06-02T07:05:17.442130Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"0 bytes copied from inbound to outbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:448","timestamp":"2021-06-02T07:05:17.442260Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"error","msg":"operation failed: migration of disk sda failed: Input/output error","pos":"qemuMigrationNBDReportMirrorError:554","subcomponent":"libvirt","thread":"31","timestamp":"2021-06-02T07:05:17.444000Z"} {"component":"virt-launcher","level":"info","msg":"0 bytes copied from inbound to outbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:448","timestamp":"2021-06-02T07:05:17.450416Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","level":"info","msg":"0 bytes copied outbound to inbound","outbound":"862d59ef-720e-4ab9-8024-199c0f0000f5-49153-source.sock","pos":"migration-proxy.go:442","timestamp":"2021-06-02T07:05:17.455747Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} {"component":"virt-launcher","kind":"","level":"info","msg":"Migration job failed","name":"windows-vm-wth-guest-tools-1622617042-0072374","namespace":"virt-readonly-disk-test-vm-with-windows-guest-tools","pos":"live-migration-source.go:696","timestamp":"2021-06-02T07:05:17.833243Z","uid":"862d59ef-720e-4ab9-8024-199c0f0000f5"} =============== VM spec ================================== apiVersion: v1 items: - apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: kubemacpool.io/transaction-timestamp: "2021-06-02T06:57:29.129022755Z" kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1alpha3 vm.kubevirt.io/validations: | [ { "name": "minimal-required-memory", "path": "jsonpath::.spec.domain.resources.requests.memory", "rule": "integer", "message": "This VM requires more memory.", "min": 536870912 }, { "name": "windows-virtio-bus", "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "rule": "enum", "message": "virto disk bus type has better performance, install virtio drivers in VM and change bus type", "values": ["virtio"], "justWarning": true }, { "name": "windows-disk-bus", "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "rule": "enum", "message": "disk bus has to be either virtio or sata or scsi", "values": ["virtio", "sata", "scsi"] }, { "name": "windows-cd-bus", "path": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus", "rule": "enum", "message": "cd bus has to be sata", "values": ["sata"] } ] creationTimestamp: "2021-06-02T06:57:26Z" finalizers: - k8s.v1.cni.cncf.io/kubeMacPool generation: 2 labels: app: windows-vm-wth-guest-tools-1622617042-0072374 vm.kubevirt.io/template: windows2k19-server-medium vm.kubevirt.io/template.namespace: openshift vm.kubevirt.io/template.revision: "1" vm.kubevirt.io/template.version: v0.14.0 name: windows-vm-wth-guest-tools-1622617042-0072374 namespace: virt-readonly-disk-test-vm-with-windows-guest-tools resourceVersion: "1216723" uid: f2fb88b9-424d-4dfb-a2b4-6178543f95f2 spec: dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: creationTimestamp: null name: windows-vm-wth-guest-tools-1622617042-0072374 spec: pvc: accessModes: - ReadWriteMany resources: requests: storage: 60Gi storageClassName: ocs-storagecluster-ceph-rbd volumeMode: Block source: pvc: name: win2k19 namespace: openshift-virtualization-os-images running: true template: metadata: annotations: vm.kubevirt.io/flavor: medium vm.kubevirt.io/os: windows2k19 vm.kubevirt.io/workload: server creationTimestamp: null labels: kubevirt.io/domain: windows-vm-wth-guest-tools-1622617042-0072374 kubevirt.io/size: medium kubevirt.io/vm: windows-vm-wth-guest-tools-1622617042-0072374 spec: domain: clock: timer: hpet: present: false hyperv: {} pit: tickPolicy: delay rtc: tickPolicy: catchup utc: {} cpu: cores: 1 model: Haswell-noTSX-IBRS sockets: 1 threads: 2 devices: disks: - disk: bus: sata name: windows-vm-wth-guest-tools-1622617042-0072374 - cdrom: bus: sata readonly: true tray: closed name: windows-guest-tools inputs: - bus: usb name: tablet type: tablet interfaces: - macAddress: 02:f4:4b:00:00:01 masquerade: {} model: e1000e name: default rng: {} features: acpi: {} apic: {} hyperv: frequencies: {} ipi: {} reenlightenment: {} relaxed: {} reset: {} runtime: {} spinlocks: spinlocks: 8191 synic: {} synictimer: direct: {} tlbflush: {} vapic: {} vpindex: {} machine: type: pc-q35-rhel8.4.0 resources: requests: memory: 4Gi evictionStrategy: LiveMigrate networks: - name: default pod: {} terminationGracePeriodSeconds: 180 volumes: - dataVolume: name: windows-vm-wth-guest-tools-1622617042-0072374 name: windows-vm-wth-guest-tools-1622617042-0072374 - containerDisk: image: registry.redhat.io/container-native-virtualization/virtio-win imagePullPolicy: IfNotPresent name: windows-guest-tools status: conditions: - lastProbeTime: null lastTransitionTime: "2021-06-02T06:58:09Z" status: "True" type: Ready created: true ready: true volumeSnapshotStatuses: - enabled: true name: windows-vm-wth-guest-tools-1622617042-0072374 - enabled: false name: windows-guest-tools reason: Volume type has no StorageClass defined kind: List metadata: resourceVersion: "" selfLink: "" =============== VMI describe ============================= Name: windows-vm-wth-guest-tools-1622617042-0072374 Namespace: virt-readonly-disk-test-vm-with-windows-guest-tools Labels: kubevirt.io/domain=windows-vm-wth-guest-tools-1622617042-0072374 kubevirt.io/migrationTargetNodeName=virt-vk-2-288t4-worker-0-bqzpd kubevirt.io/nodeName=virt-vk-2-288t4-worker-0-x48zb kubevirt.io/size=medium kubevirt.io/vm=windows-vm-wth-guest-tools-1622617042-0072374 Annotations: kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1alpha3 vm.kubevirt.io/flavor: medium vm.kubevirt.io/os: windows2k19 vm.kubevirt.io/workload: server API Version: kubevirt.io/v1 Kind: VirtualMachineInstance Metadata: Creation Timestamp: 2021-06-02T06:57:46Z Finalizers: foregroundDeleteVirtualMachine Generation: 30 Managed Fields: API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubevirt.io/latest-observed-api-version: f:kubevirt.io/storage-observed-api-version: f:vm.kubevirt.io/flavor: f:vm.kubevirt.io/os: f:vm.kubevirt.io/workload: f:labels: .: f:kubevirt.io/domain: f:kubevirt.io/migrationTargetNodeName: f:kubevirt.io/nodeName: f:kubevirt.io/size: f:kubevirt.io/vm: f:ownerReferences: .: k:{"uid":"f2fb88b9-424d-4dfb-a2b4-6178543f95f2"}: .: f:apiVersion: f:blockOwnerDeletion: f:controller: f:kind: f:name: f:uid: f:spec: .: f:domain: .: f:clock: .: f:timer: .: f:hpet: .: f:present: f:hyperv: .: f:present: f:pit: .: f:tickPolicy: f:rtc: .: f:tickPolicy: f:utc: f:cpu: .: f:cores: f:model: f:sockets: f:threads: f:devices: .: f:disks: f:inputs: f:interfaces: f:rng: f:features: .: f:acpi: .: f:enabled: f:apic: .: f:enabled: f:hyperv: .: f:frequencies: .: f:enabled: f:ipi: .: f:enabled: f:reenlightenment: .: f:enabled: f:relaxed: .: f:enabled: f:reset: .: f:enabled: f:runtime: .: f:enabled: f:spinlocks: .: f:spinlocks: f:synic: .: f:enabled: f:synictimer: .: f:direct: .: f:enabled: f:tlbflush: .: f:enabled: f:vapic: .: f:enabled: f:vpindex: .: f:enabled: f:firmware: .: f:uuid: f:machine: .: f:type: f:resources: .: f:requests: .: f:memory: f:evictionStrategy: f:networks: f:terminationGracePeriodSeconds: f:volumes: f:status: .: f:activePods: .: f:156c0142-278b-4ad7-ab88-95f5b50ba39a: f:d721d07b-dad6-4078-9365-296b7aeb0b86: f:guestOSInfo: f:launcherContainerImageVersion: f:migrationState: .: f:migrationUid: f:sourceNode: f:targetNode: f:targetPod: f:nodeName: f:qosClass: Manager: virt-controller Operation: Update Time: 2021-06-02T07:00:17Z API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:interfaces: f:migrationMethod: f:migrationState: f:completed: f:endTimestamp: f:failed: f:mode: f:startTimestamp: f:targetDirectMigrationNodePorts: .: f:36663: f:43165: f:44149: f:targetNodeAddress: f:phase: f:volumeStatus: Manager: virt-handler Operation: Update Time: 2021-06-02T07:24:58Z Owner References: API Version: kubevirt.io/v1 Block Owner Deletion: true Controller: true Kind: VirtualMachine Name: windows-vm-wth-guest-tools-1622617042-0072374 UID: f2fb88b9-424d-4dfb-a2b4-6178543f95f2 Resource Version: 1242664 UID: 862d59ef-720e-4ab9-8024-199c0f0000f5 Spec: Domain: Clock: Timer: Hpet: Present: false Hyperv: Present: true Pit: Present: true Tick Policy: delay Rtc: Present: true Tick Policy: catchup Utc: Cpu: Cores: 1 Model: Haswell-noTSX-IBRS Sockets: 1 Threads: 2 Devices: Disks: Disk: Bus: sata Name: windows-vm-wth-guest-tools-1622617042-0072374 Cdrom: Bus: sata Readonly: true Tray: closed Name: windows-guest-tools Inputs: Bus: usb Name: tablet Type: tablet Interfaces: Mac Address: 02:f4:4b:00:00:01 Masquerade: Model: e1000e Name: default Rng: Features: Acpi: Enabled: true Apic: Enabled: true Hyperv: Frequencies: Enabled: true Ipi: Enabled: true Reenlightenment: Enabled: true Relaxed: Enabled: true Reset: Enabled: true Runtime: Enabled: true Spinlocks: Enabled: true Spinlocks: 8191 Synic: Enabled: true Synictimer: Direct: Enabled: true Enabled: true Tlbflush: Enabled: true Vapic: Enabled: true Vpindex: Enabled: true Firmware: Uuid: 0ce8cd97-9d53-5e9f-b145-b9f0fd9d6af0 Machine: Type: pc-q35-rhel8.4.0 Resources: Requests: Cpu: 100m Memory: 4Gi Eviction Strategy: LiveMigrate Networks: Name: default Pod: Termination Grace Period Seconds: 180 Volumes: Data Volume: Name: windows-vm-wth-guest-tools-1622617042-0072374 Name: windows-vm-wth-guest-tools-1622617042-0072374 Container Disk: Image: registry.redhat.io/container-native-virtualization/virtio-win Image Pull Policy: IfNotPresent Name: windows-guest-tools Status: Active Pods: 156c0142-278b-4ad7-ab88-95f5b50ba39a: virt-vk-2-288t4-worker-0-x48zb d721d07b-dad6-4078-9365-296b7aeb0b86: virt-vk-2-288t4-worker-0-bqzpd Conditions: Last Probe Time: <nil> Last Transition Time: <nil> Status: True Type: LiveMigratable Last Probe Time: <nil> Last Transition Time: 2021-06-02T06:58:09Z Status: True Type: Ready Last Probe Time: 2021-06-02T06:59:49Z Last Transition Time: <nil> Status: True Type: AgentConnected Guest OS Info: Interfaces: Interface Name: Ethernet 3 Ip Address: 10.128.2.132 Ip Addresses: 10.128.2.132 Mac: 02:f4:4b:00:00:01 Name: default Interface Name: Loopback Pseudo-Interface 1 Ip Address: 127.0.0.1 Ip Addresses: ::1 127.0.0.1 Launcher Container Image Version: registry.redhat.io/container-native-virtualization/virt-launcher@sha256:e4f055c67b16dfbf2298d0e805e4387dcdce080f169d49579715b2e04bf13fde Migration Method: BlockMigration Migration State: Completed: true End Timestamp: 2021-06-02T07:05:17Z Failed: true Migration UID: 4238e03e-84a8-4e3c-963c-3e6e52f3a91d Mode: PreCopy Source Node: virt-vk-2-288t4-worker-0-x48zb Start Timestamp: 2021-06-02T07:00:17Z Target Direct Migration Node Ports: 36663: 0 43165: 49152 44149: 49153 Target Node: virt-vk-2-288t4-worker-0-bqzpd Target Node Address: 10.129.2.5 Target Pod: virt-launcher-windows-vm-wth-guest-tools-1622617042-007237vwvgw Node Name: virt-vk-2-288t4-worker-0-x48zb Phase: Running Qos Class: Burstable Volume Status: Name: windows-guest-tools Target: sdb Name: windows-vm-wth-guest-tools-1622617042-0072374 Target: sda Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 28m disruptionbudget-controller Created PodDisruptionBudget kubevirt-disruption-budget-wn8hh Normal SuccessfulCreate 28m virtualmachine-controller Created virtual machine pod virt-launcher-windows-vm-wth-guest-tools-1622617042-007237djn6t Normal Started 28m virt-handler VirtualMachineInstance started. Normal PreparingTarget 26m virt-handler Migration Target is listening at 10.129.2.5, on ports: 36663,43165,44149 Normal PreparingTarget 26m (x2 over 26m) virt-handler VirtualMachineInstance Migration Target Prepared. Normal Migrating 22m (x11 over 26m) virt-handler VirtualMachineInstance is migrating. Normal Created 2m4s (x41 over 28m) virt-handler VirtualMachineInstance defined.
OCP version: 4.8.5 virt operator version: virt-operator-container-v4.8.1-2 RHEL VM tested in: https://bugzilla.redhat.com/show_bug.cgi?id=1927378 Windows VM: tested with BM 3 times migration average time is 3-4 min with cdrom. VM spec: apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: kubemacpool.io/transaction-timestamp: '2021-08-19T14:34:48.881840969Z' kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1alpha3 name.os.template.kubevirt.io/win10: Microsoft Windows 10 vm.kubevirt.io/validations: | [ { "name": "minimal-required-memory", "path": "jsonpath::.spec.domain.resources.requests.memory", "rule": "integer", "message": "This VM requires more memory.", "min": 2147483648 }, { "name": "windows-virtio-bus", "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "rule": "enum", "message": "virto disk bus type has better performance, install virtio drivers in VM and change bus type", "values": ["virtio"], "justWarning": true }, { "name": "windows-disk-bus", "path": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].disk.bus", "rule": "enum", "message": "disk bus has to be either virtio or sata or scsi", "values": ["virtio", "sata", "scsi"] }, { "name": "windows-cd-bus", "path": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus", "valid": "jsonpath::.spec.domain.devices.disks[*].cdrom.bus", "rule": "enum", "message": "cd bus has to be sata", "values": ["sata"] } ] resourceVersion: '231804' name: test-cdrom-mig uid: ad01944d-2c1a-49e5-a762-003a643803c7 creationTimestamp: '2021-08-19T14:34:48Z' generation: 1 managedFields: - apiVersion: kubevirt.io/v1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:name.os.template.kubevirt.io/win10': {} 'f:vm.kubevirt.io/validations': {} 'f:labels': .: {} 'f:app': {} 'f:os.template.kubevirt.io/win10': {} 'f:vm.kubevirt.io/template': {} 'f:vm.kubevirt.io/template.namespace': {} 'f:vm.kubevirt.io/template.revision': {} 'f:vm.kubevirt.io/template.version': {} 'f:workload.template.kubevirt.io/desktop': {} 'f:spec': .: {} 'f:dataVolumeTemplates': {} 'f:running': {} 'f:template': .: {} 'f:metadata': .: {} 'f:annotations': {} 'f:labels': {} 'f:spec': .: {} 'f:domain': .: {} 'f:clock': .: {} 'f:timer': .: {} 'f:hpet': .: {} 'f:present': {} 'f:hyperv': {} 'f:pit': .: {} 'f:tickPolicy': {} 'f:rtc': .: {} 'f:tickPolicy': {} 'f:utc': {} 'f:cpu': .: {} 'f:cores': {} 'f:sockets': {} 'f:threads': {} 'f:devices': .: {} 'f:disks': {} 'f:inputs': {} 'f:interfaces': {} 'f:features': .: {} 'f:acpi': {} 'f:apic': {} 'f:hyperv': 'f:runtime': {} 'f:vpindex': {} 'f:frequencies': {} 'f:vapic': {} 'f:reenlightenment': {} .: {} 'f:tlbflush': {} 'f:synic': {} 'f:ipi': {} 'f:spinlocks': .: {} 'f:spinlocks': {} 'f:synictimer': .: {} 'f:direct': {} 'f:relaxed': {} 'f:reset': {} 'f:machine': .: {} 'f:type': {} 'f:resources': .: {} 'f:requests': .: {} 'f:memory': {} 'f:evictionStrategy': {} 'f:hostname': {} 'f:networks': {} 'f:terminationGracePeriodSeconds': {} 'f:volumes': {} manager: Mozilla operation: Update time: '2021-08-19T14:34:48Z' - apiVersion: kubevirt.io/v1alpha3 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': 'f:kubevirt.io/latest-observed-api-version': {} 'f:kubevirt.io/storage-observed-api-version': {} 'f:status': .: {} 'f:conditions': {} 'f:created': {} 'f:ready': {} 'f:volumeSnapshotStatuses': {} manager: virt-controller operation: Update time: '2021-08-19T14:39:08Z' namespace: test-win-migration finalizers: - k8s.v1.cni.cncf.io/kubeMacPool labels: app: test-cdrom-mig os.template.kubevirt.io/win10: 'true' vm.kubevirt.io/template: windows10-desktop-large vm.kubevirt.io/template.namespace: openshift vm.kubevirt.io/template.revision: '1' vm.kubevirt.io/template.version: v0.15.0 workload.template.kubevirt.io/desktop: 'true' spec: dataVolumeTemplates: - metadata: creationTimestamp: null name: test-cdrom-mig-rootdisk-71sr2 spec: pvc: accessModes: - ReadWriteMany resources: requests: storage: 60Gi storageClassName: ocs-storagecluster-ceph-rbd volumeMode: Block source: http: url: >- http://cnv-qe-server.rhevdev.lab.eng.rdu2.redhat.com/files/cnv-tests/windows-images/win_10.qcow2 running: true template: metadata: annotations: vm.kubevirt.io/flavor: large vm.kubevirt.io/os: windows10 vm.kubevirt.io/workload: desktop creationTimestamp: null labels: kubevirt.io/domain: test-cdrom-mig kubevirt.io/size: large os.template.kubevirt.io/win10: 'true' vm.kubevirt.io/name: test-cdrom-mig workload.template.kubevirt.io/desktop: 'true' spec: domain: clock: timer: hpet: present: false hyperv: {} pit: tickPolicy: delay rtc: tickPolicy: catchup utc: {} cpu: cores: 4 sockets: 1 threads: 1 devices: disks: - cdrom: bus: sata name: windows-guest-tools - bootOrder: 1 disk: bus: sata name: rootdisk inputs: - bus: usb name: tablet type: tablet interfaces: - macAddress: '02:f4:ed:00:00:03' masquerade: {} model: e1000e name: default features: acpi: {} apic: {} hyperv: reenlightenment: {} ipi: {} synic: {} synictimer: direct: {} spinlocks: spinlocks: 8191 reset: {} relaxed: {} vpindex: {} runtime: {} tlbflush: {} frequencies: {} vapic: {} machine: type: pc-q35-rhel8.4.0 resources: requests: memory: 10Gi evictionStrategy: LiveMigrate hostname: test-cdrom-mig networks: - name: default pod: {} terminationGracePeriodSeconds: 3600 volumes: - containerDisk: image: >- registry.redhat.io/container-native-virtualization/virtio-win@sha256:013fc43d8ff95d5a874ea9e512483d43ac1ec6d80026f5dcfdf12f882648ad64 name: windows-guest-tools - dataVolume: name: test-cdrom-mig-rootdisk-71sr2 name: rootdisk status: conditions: - lastProbeTime: null lastTransitionTime: '2021-08-19T14:39:03Z' status: 'True' type: Ready created: true ready: true volumeSnapshotStatuses: - enabled: false name: windows-guest-tools reason: Volume type has no StorageClass defined - enabled: true name: rootdisk Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 25m virtualmachine-controller Created virtual machine pod virt-launcher-test-cdrom-mig-s74pc Normal SuccessfulCreate 25m disruptionbudget-controller Created PodDisruptionBudget kubevirt-disruption-budget-zqxnk Normal Started 24m virt-handler VirtualMachineInstance started. Normal SuccessfulCreate 22m virtualmachine-controller Created PodDisruptionBudget kubevirt-migration-pdb-test-cdrom-mig-migration-cs668 Normal PreparingTarget 21m virt-handler Migration Target is listening at 10.131.0.4, on ports: 34669,35071,34611 Normal Migrating 19m (x7 over 21m) virt-handler VirtualMachineInstance is migrating. Normal SuccessfulDelete 17m disruptionbudget-controller Deleted PodDisruptionBudget kubevirt-migration-pdb-test-cdrom-mig-migration-cs668 Normal SuccessfulCreate 16m virtualmachine-controller Created PodDisruptionBudget kubevirt-migration-pdb-test-cdrom-mig-migration-p42gl Normal Created 16m (x14 over 17m) virt-handler VirtualMachineInstance defined. Normal Migrating 15m (x9 over 16m) virt-handler VirtualMachineInstance is migrating. Normal SuccessfulDelete 12m disruptionbudget-controller Deleted PodDisruptionBudget kubevirt-migration-pdb-test-cdrom-mig-migration-p42gl Normal Created 3m39s (x48 over 24m) virt-handler VirtualMachineInstance defined. Normal SuccessfulCreate 3m15s virtualmachine-controller Created PodDisruptionBudget kubevirt-migration-pdb-test-cdrom-mig-migration-j8gg4 Normal PreparingTarget 2m54s (x3 over 21m) virt-handler VirtualMachineInstance Migration Target Prepared. Normal PreparingTarget 2m54s virt-handler Migration Target is listening at 10.131.0.4, on ports: 36905,35781,40763 ipinto@localhost ~ oc get pods test-cdrom-mig -n test-win-migration Error from server (NotFound): pods "test-cdrom-mig" not found ✘ ipinto@localhost ~ oc get pods -n test-win-migration NAME READY STATUS RESTARTS AGE virt-launcher-test-cdrom-mig-6lp4l 2/2 Running 0 4m58s virt-launcher-test-cdrom-mig-cnszj 0/2 Completed 0 24m virt-launcher-test-cdrom-mig-s74pc 0/2 Completed 0 26m virt-launcher-test-cdrom-mig-szf6b 0/2 Completed 0 18m moving to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.8.1 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3259