Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Israel,
I'm not sure exactly what question you're answering in comment 6. I was already aware that this is specific to iSCSI, not NFS.
But at this point, I'm not sure what the RHV operations you describe in comment 0 translate to in terms of lower level operations which is where the problem (probably) lies.
So I'm looking for more details on what the reproducer here is in terms of the actions RHV performs at the libvirt and qemu levels.
In particular, I'm not sure exactly what "suspend" and "resume" mean in this case. Are they using within-guest suspend-to-RAM operations, or are they doing an external suspend of the VM (e.g. with a qemu stop).
(In reply to David Gibson from comment #8)
> So I'm looking for more details on what the reproducer here is in terms of
> the actions RHV performs at the libvirt and qemu levels.
>
> In particular, I'm not sure exactly what "suspend" and "resume" mean in this
> case. Are they using within-guest suspend-to-RAM operations, or are they
> doing an external suspend of the VM (e.g. with a qemu stop).
In terms of libvirt operations, we use 'save' and 'restore' (in some cases 'restoreFlags' but that should be similar). No interaction with the guest is made.
Thanks Arik,
Andrea, can you fill me in on what the 'save' and 'restore' libvirt operations will translate to in terms of qemu actions?
Comment 12Andrea Bolognani
2018-07-10 11:41:37 UTC
(In reply to David Gibson from comment #11)
> Andrea, can you fill me in on what the 'save' and 'restore' libvirt
> operations will translate to in terms of qemu actions?
'save' is QMP 'stop' followed by QMP 'migrate' with an FD as
destination; calling 'restore' results in libvirt calling QEMU
with '-incoming' (and possibly QMP 'migrate-incoming' later,
if '-incoming defer' is supported).
Basically it's a regular migration, but to / from a file.
There are some I/O errors in the attachment 1450917[details] , file vdsm.log:
read failed after 0 of 65536 at 53697511424: Input/output error
WARNING: Error counts reached a limit of 3. Device /dev/mapper/36090a06890b4b694996615b6232b7ce7 was disabled
Perhaps the problem comes from there?
(In reply to Laurent Vivier from comment #13)
> There are some I/O errors in the attachment 1450917[details] , file
> vdsm.log:
>
> read failed after 0 of 65536 at 53697511424: Input/output error
> WARNING: Error counts reached a limit of 3. Device
> /dev/mapper/36090a06890b4b694996615b6232b7ce7 was disabled
>
> Perhaps the problem comes from there?
Yes it can happen since it, I tested it now with new setup.
I not reproduce on ISCSI.
(In reply to Israel Pinto from comment #14)
> (In reply to Laurent Vivier from comment #13)
> > There are some I/O errors in the attachment 1450917[details] , file
> > vdsm.log:
> >
> > read failed after 0 of 65536 at 53697511424: Input/output error
> > WARNING: Error counts reached a limit of 3. Device
> > /dev/mapper/36090a06890b4b694996615b6232b7ce7 was disabled
> >
> > Perhaps the problem comes from there?
>
> Yes it can happen since it, I tested it now with new setup.
> It not reproduce on ISCSI.
typo error
Israel,
To clarify, in comment 14 are you saying that you're no longer able to reproduce the problem, so the working theory is that it was caused by a transient network problem?
i.e. can we close this as NOTABUG?
(In reply to David Gibson from comment #16)
> Israel,
>
> To clarify, in comment 14 are you saying that you're no longer able to
> reproduce the problem, so the working theory is that it was caused by a
> transient network problem?
>
> i.e. can we close this as NOTABUG?
Yes it looks like network problem.
Closing the BZ.
Description of problem: On rhvm environment we hotplug memory to VM and suspend it, and resume VM. The VM is up after about 40 min. Version-Release number of selected component (if applicable): OS Version:RHEL - 7.5 - 8.el7 OS Description: Red Hat Enterprise Linux Server 7.5 (Maipo) Kernel Version:3.10.0 - 862.6.1.el7.ppc64le KVM Version:2.10.0 - 21.el7_5.3 LIBVIRT Version:libvirt-3.9.0-14.el7_5.6 VDSM Version:vdsm-4.20.30-1.el7ev libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.ppc64le qemu-kvm-common-rhev-2.10.0-21.el7_5.3.ppc64le ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch qemu-img-rhev-2.10.0-21.el7_5.3.ppc64le qemu-kvm-rhev-2.10.0-21.el7_5.3.ppc64le How reproducible: 100% Steps to Reproduce: On PPC64 1. Create vm with OS on ISCSI disk 2. Start VM 3. Hotplug memory to vm 4. Suspend VM 5. Resume VM Actual results: VM is up after about 40 min, VM is in restore state. With NFS storage vm starts right away. Additional info: Start VM action on vdsm log: [root@ibm-p8-rhevm-hv-02 vdsm]# grep -ir flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d vdsm.log:2018-06-13 13:06:42,193+0300 INFO (jsonrpc/6) [api.virt] START create(vmParams={u'xml': u'<?xml version="1.0" encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"><name>memory_hotplug_test</name><uuid>a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63</uuid><memory>3145728</memory><currentMemory>3145728</currentMemory><maxMemory slots="16">16777216</maxMemory><vcpu current="1">16</vcpu><clock offset="variable" adjustment="0"><timer name="rtc" tickpolicy="catchup"></timer><timer name="pit" tickpolicy="delay"></timer></clock><cpu mode="host-model"><model>power8</model><topology cores="1" threads="1" sockets="16"></topology><numa><cell id="0" cpus="0" memory="3145728"></cell></numa></cpu><cputune></cputune><devices><input type="tablet" bus="usb"></input><channel type="unix"><target type="virtio" name="ovirt-guest-agent.0"></target><source mode="bind" path="/var/lib/libvirt/qemu/channels/a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63.ovirt-guest-agent.0"></source></channel><channel type="unix"><target type="virtio" name="org.qemu.guest_agent.0"></target><source mode="bind" path="/var/lib/libvirt/qemu/channels/a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63.org.qemu.guest_agent.0"></source></channel><emulator text="/usr/bin/qemu-system-ppc64"></emulator><controller type="scsi" index="0"><address type="spapr-vio"></address></controller><rng model="virtio"><backend model="random">/dev/urandom</backend><alias name="ua-2a267834-ea6b-4a26-94ca-36ab3f22e63a"></alias></rng><controller type="virtio-serial" index="0" ports="16"><alias name="ua-5f3d68a7-ea48-4086-9375-3e82ea967e2e"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"></address></controller><controller type="usb" model="nec-xhci" index="0"><address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"></address></controller><controller type="scsi" model="virtio-scsi" index="1"><alias name="ua-c097c02c-f893-4455-be81-4c0f1f355e5f"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"></address></controller><graphics type="vnc" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" keymap="en-us"><listen type="network" network="vdsm-ovirtmgmt"></listen></graphics><video><model type="vga" vram="16384" heads="1"></model><alias name="ua-c8f963db-f64a-4acb-8d2d-2a664125061e"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x07" type="pci"></address></video><memballoon model="none"></memballoon><interface type="bridge"><model type="virtio"></model><link state="up"></link><source bridge="ovirtmgmt"></source><alias name="ua-7ce5c17d-568f-40e7-abf9-38f2ffc1e363"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x01" type="pci"></address><mac address="00:1a:4a:16:01:a7"></mac><filterref filter="vdsm-no-mac-spoofing"></filterref><bandwidth></bandwidth></interface><disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"></driver><source file="" startupPolicy="optional"></source><target dev="sdc" bus="scsi"></target><readonly></readonly><alias name="ua-fb4fbcb3-9589-4aac-9117-bba286811002"></alias><address bus="0" controller="0" unit="2" type="drive" target="0"></address></disk><disk snapshot="no" type="block" device="disk"><target dev="vda" bus="virtio"></target><source dev="/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11"></source><driver name="qemu" io="native" type="qcow2" error_policy="stop" cache="none"></driver><alias name="ua-c2bd74ad-dfab-416e-b366-42b401de204c"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x05" type="pci"></address><boot order="1"></boot><serial>c2bd74ad-dfab-416e-b366-42b401de204c</serial></disk></devices><os><type arch="ppc64" machine="pseries-rhel7.5.0">hvm</type></os><metadata><ovirt-tune:qos></ovirt-tune:qos><ovirt-vm:vm><minGuaranteedMemoryMb type="int">3072</minGuaranteedMemoryMb><clusterVersion>4.2</clusterVersion><ovirt-vm:custom></ovirt-vm:custom><ovirt-vm:device mac_address="00:1a:4a:16:01:a7"><ovirt-vm:custom></ovirt-vm:custom></ovirt-vm:device><ovirt-vm:device devtype="disk" name="vda"><ovirt-vm:poolID>74170a2d-821a-402d-8311-8e1cd3cac00a</ovirt-vm:poolID><ovirt-vm:volumeID>393fb11b-f19c-4299-9cd9-cf05b299ee11</ovirt-vm:volumeID><ovirt-vm:imageID>c2bd74ad-dfab-416e-b366-42b401de204c</ovirt-vm:imageID><ovirt-vm:domainID>3543271c-284b-461c-be84-8c69b4be7a11</ovirt-vm:domainID></ovirt-vm:device><launchPaused>false</launchPaused><resumeBehavior>auto_resume</resumeBehavior></ovirt-vm:vm></metadata></domain>', u'memoryDumpVolume': {u'imageID': u'4534b792-1c84-40de-a796-9299572bdf50', u'volumeID': u'd0965e4f-3d35-4bf0-bb55-0c3c3b1d4aa3', u'domainID': u'3543271c-284b-461c-be84-8c69b4be7a11', u'poolID': u'74170a2d-821a-402d-8311-8e1cd3cac00a'}, u'memoryConfVolume': {u'imageID': u'e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea', u'volumeID': u'6e5065fc-7c85-4084-956a-99364c0357d5', u'domainID': u'3543271c-284b-461c-be84-8c69b4be7a11', u'poolID': u'74170a2d-821a-402d-8311-8e1cd3cac00a'}}) from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, vmId= (api:46) vdsm.log:2018-06-13 13:06:42,193+0300 INFO (jsonrpc/6) [vdsm.api] START prepareImage(sdUUID=u'3543271c-284b-461c-be84-8c69b4be7a11', spUUID=u'74170a2d-821a-402d-8311-8e1cd3cac00a', imgUUID=u'e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea', leafUUID=u'6e5065fc-7c85-4084-956a-99364c0357d5', allowIllegal=False) from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, task_id=7e0ce949-0160-490b-91f3-56da20a3a147 (api:46) vdsm.log:2018-06-13 13:06:42,971+0300 INFO (jsonrpc/6) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea/6e5065fc-7c85-4084-956a-99364c0357d5', 'type': 'block'}, 'path': u'/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea/6e5065fc-7c85-4084-956a-99364c0357d5', 'imgVolumesInfo': [{'domainID': u'3543271c-284b-461c-be84-8c69b4be7a11', 'leaseOffset': 114294784, 'path': u'/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea/6e5065fc-7c85-4084-956a-99364c0357d5', 'volumeID': '6e5065fc-7c85-4084-956a-99364c0357d5', 'leasePath': '/dev/3543271c-284b-461c-be84-8c69b4be7a11/leases', 'imageID': u'e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea'}]} from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, task_id=7e0ce949-0160-490b-91f3-56da20a3a147 (api:52) vdsm.log:2018-06-13 13:06:42,980+0300 INFO (jsonrpc/6) [vdsm.api] START teardownImage(sdUUID=u'3543271c-284b-461c-be84-8c69b4be7a11', spUUID=u'74170a2d-821a-402d-8311-8e1cd3cac00a', imgUUID=u'e68b53cc-18f5-4b43-ab26-40bbd1c5f7ea', volUUID=None) from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, task_id=c649a069-fe96-4ecd-b4d4-1c94c5abecdf (api:46) vdsm.log:2018-06-13 13:06:43,537+0300 INFO (jsonrpc/6) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, task_id=c649a069-fe96-4ecd-b4d4-1c94c5abecdf (api:52) vdsm.log:2018-06-13 13:06:43,566+0300 INFO (jsonrpc/6) [api.virt] FINISH create return={'status': {'message': 'Done', 'code': 0}, 'vmList': {'username': u'None', 'acpiEnable': 'false', 'emulatedMachine': 'pseries-rhel7.5.0', 'tabletEnable': 'true', 'vmId': 'a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63', 'guestDiskMapping': {}, 'timeOffset': '1', 'cpuType': 'power8', 'smp': '1', 'guestNumaNodes': [{'nodeIndex': 0, 'cpus': '0', 'memory': '1024'}], u'xml': '<domain type=\'kvm\' id=\'119\'>\n <name>memory_hotplug_test</name>\n <uuid>a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63</uuid>\n <metadata xmlns:ns0="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0">\n <ns0:qos/>\n <ovirt-vm:vm xmlns:ovirt-vm="http://ovirt.org/vm/1.0">\n <ovirt-vm:clusterVersion>4.2</ovirt-vm:clusterVersion>\n <ovirt-vm:destroy_on_reboot type="bool">True</ovirt-vm:destroy_on_reboot>\n <ovirt-vm:guestAgentAPIVersion type="int">3</ovirt-vm:guestAgentAPIVersion>\n <ovirt-vm:launchPaused>false</ovirt-vm:launchPaused>\n <ovirt-vm:memGuaranteedSize type="int">512</ovirt-vm:memGuaranteedSize>\n <ovirt-vm:minGuaranteedMemoryMb type="int">512</ovirt-vm:minGuaranteedMemoryMb>\n <ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior>\n <ovirt-vm:startTime type="float">1528876556.91</ovirt-vm:startTime>\n <ovirt-vm:device mac_address="00:1a:4a:16:01:a7">\n <ovirt-vm:network>ovirtmgmt</ovirt-vm:network>\n <ovirt-vm:specParams/>\n <ovirt-vm:vm_custom/>\n </ovirt-vm:device>\n <ovirt-vm:device devtype="disk" name="vda">\n <ovirt-vm:domainID>3543271c-284b-461c-be84-8c69b4be7a11</ovirt-vm:domainID>\n <ovirt-vm:guestName>/dev/vda</ovirt-vm:guestName>\n <ovirt-vm:imageID>c2bd74ad-dfab-416e-b366-42b401de204c</ovirt-vm:imageID>\n <ovirt-vm:poolID>74170a2d-821a-402d-8311-8e1cd3cac00a</ovirt-vm:poolID>\n <ovirt-vm:volumeID>393fb11b-f19c-4299-9cd9-cf05b299ee11</ovirt-vm:volumeID>\n <ovirt-vm:specParams/>\n <ovirt-vm:vm_custom/>\n <ovirt-vm:volumeChain>\n <ovirt-vm:volumeChainNode>\n <ovirt-vm:domainID>3543271c-284b-461c-be84-8c69b4be7a11</ovirt-vm:domainID>\n <ovirt-vm:imageID>c2bd74ad-dfab-416e-b366-42b401de204c</ovirt-vm:imageID>\n <ovirt-vm:leaseOffset type="int">112197632</ovirt-vm:leaseOffset>\n <ovirt-vm:leasePath>/dev/3543271c-284b-461c-be84-8c69b4be7a11/leases</ovirt-vm:leasePath>\n <ovirt-vm:path>/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11</ovirt-vm:path>\n <ovirt-vm:volumeID>393fb11b-f19c-4299-9cd9-cf05b299ee11</ovirt-vm:volumeID>\n </ovirt-vm:volumeChainNode>\n <ovirt-vm:volumeChainNode>\n <ovirt-vm:domainID>3543271c-284b-461c-be84-8c69b4be7a11</ovirt-vm:domainID>\n <ovirt-vm:imageID>c2bd74ad-dfab-416e-b366-42b401de204c</ovirt-vm:imageID>\n <ovirt-vm:leaseOffset type="int">109051904</ovirt-vm:leaseOffset>\n <ovirt-vm:leasePath>/dev/3543271c-284b-461c-be84-8c69b4be7a11/leases</ovirt-vm:leasePath>\n <ovirt-vm:path>/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/d693d106-ae01-41a5-9e70-562369d838dd</ovirt-vm:path>\n <ovirt-vm:volumeID>d693d106-ae01-41a5-9e70-562369d838dd</ovirt-vm:volumeID>\n </ovirt-vm:volumeChainNode>\n </ovirt-vm:volumeChain>\n </ovirt-vm:device>\n <ovirt-vm:device devtype="disk" name="sdc">\n <ovirt-vm:specParams/>\n <ovirt-vm:vm_custom/>\n </ovirt-vm:device>\n</ovirt-vm:vm>\n </metadata>\n <maxMemory slots=\'16\' unit=\'KiB\'>16777216</maxMemory>\n <memory unit=\'KiB\'>3145728</memory>\n <currentMemory unit=\'KiB\'>3145728</currentMemory>\n <vcpu placement=\'static\' current=\'1\'>16</vcpu>\n <resource>\n <partition>/machine</partition>\n </resource>\n <os>\n <type arch=\'ppc64\' machine=\'pseries-rhel7.5.0\'>hvm</type>\n </os>\n <cpu mode=\'host-model\' check=\'partial\'>\n <model fallback=\'forbid\'>power8</model>\n <topology sockets=\'16\' cores=\'1\' threads=\'1\'/>\n <numa>\n <cell id=\'0\' cpus=\'0\' memory=\'1048576\' unit=\'KiB\'/>\n </numa>\n </cpu>\n <clock offset=\'variable\' adjustment=\'1\' basis=\'utc\'>\n <timer name=\'rtc\' tickpolicy=\'catchup\'/>\n <timer name=\'pit\' tickpolicy=\'delay\'/>\n </clock>\n <on_poweroff>destroy</on_poweroff>\n <on_reboot>restart</on_reboot>\n <on_crash>destroy</on_crash>\n <devices>\n <emulator>/usr/libexec/qemu-kvm</emulator>\n <disk type=\'file\' device=\'cdrom\'>\n <driver name=\'qemu\' type=\'raw\' error_policy=\'report\'/>\n <source startupPolicy=\'optional\'/>\n <target dev=\'sdc\' bus=\'scsi\'/>\n <readonly/>\n <alias name=\'ua-fb4fbcb3-9589-4aac-9117-bba286811002\'/>\n <address type=\'drive\' controller=\'0\' bus=\'0\' target=\'0\' unit=\'2\'/>\n </disk>\n <disk type=\'block\' device=\'disk\' snapshot=\'no\'>\n <driver name=\'qemu\' type=\'qcow2\' cache=\'none\' error_policy=\'stop\' io=\'native\'/>\n <source dev=\'/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11\'/>\n <backingStore type=\'block\' index=\'1\'>\n <format type=\'qcow2\'/>\n <source dev=\'/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/d693d106-ae01-41a5-9e70-562369d838dd\'/>\n <backingStore/>\n </backingStore>\n <target dev=\'vda\' bus=\'virtio\'/>\n <serial>c2bd74ad-dfab-416e-b366-42b401de204c</serial>\n <boot order=\'1\'/>\n <alias name=\'ua-c2bd74ad-dfab-416e-b366-42b401de204c\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x05\' function=\'0x0\'/>\n </disk>\n <controller type=\'scsi\' index=\'0\'>\n <alias name=\'scsi0\'/>\n <address type=\'spapr-vio\' reg=\'0x2000\'/>\n </controller>\n <controller type=\'scsi\' index=\'1\' model=\'virtio-scsi\'>\n <alias name=\'ua-c097c02c-f893-4455-be81-4c0f1f355e5f\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x0\'/>\n </controller>\n <controller type=\'virtio-serial\' index=\'0\' ports=\'16\'>\n <alias name=\'ua-5f3d68a7-ea48-4086-9375-3e82ea967e2e\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x0\'/>\n </controller>\n <controller type=\'usb\' index=\'0\' model=\'nec-xhci\'>\n <alias name=\'usb\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x04\' function=\'0x0\'/>\n </controller>\n <controller type=\'pci\' index=\'0\' model=\'pci-root\'>\n <model name=\'spapr-pci-host-bridge\'/>\n <target index=\'0\'/>\n <alias name=\'pci.0\'/>\n </controller>\n <interface type=\'bridge\'>\n <mac address=\'00:1a:4a:16:01:a7\'/>\n <source bridge=\'ovirtmgmt\'/>\n <target dev=\'vnet0\'/>\n <model type=\'virtio\'/>\n <filterref filter=\'vdsm-no-mac-spoofing\'/>\n <link state=\'up\'/>\n <alias name=\'ua-7ce5c17d-568f-40e7-abf9-38f2ffc1e363\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x0\'/>\n </interface>\n <channel type=\'unix\'>\n <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63.ovirt-guest-agent.0\'/>\n <target type=\'virtio\' name=\'ovirt-guest-agent.0\' state=\'connected\'/>\n <alias name=\'channel0\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'1\'/>\n </channel>\n <channel type=\'unix\'>\n <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63.org.qemu.guest_agent.0\'/>\n <target type=\'virtio\' name=\'org.qemu.guest_agent.0\' state=\'connected\'/>\n <alias name=\'channel1\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'2\'/>\n </channel>\n <input type=\'tablet\' bus=\'usb\'>\n <alias name=\'input0\'/>\n <address type=\'usb\' bus=\'0\' port=\'1\'/>\n </input>\n <input type=\'keyboard\' bus=\'usb\'>\n <alias name=\'input1\'/>\n <address type=\'usb\' bus=\'0\' port=\'2\'/>\n </input>\n <input type=\'mouse\' bus=\'usb\'>\n <alias name=\'input2\'/>\n <address type=\'usb\' bus=\'0\' port=\'3\'/>\n </input>\n <graphics type=\'vnc\' port=\'5900\' autoport=\'yes\' listen=\'10.16.160.29\' keymap=\'en-us\' passwdValidTo=\'1970-01-01T00:00:01\'>\n <listen type=\'network\' address=\'10.16.160.29\' network=\'vdsm-ovirtmgmt\'/>\n </graphics>\n <video>\n <model type=\'vga\' vram=\'16384\' heads=\'1\' primary=\'yes\'/>\n <alias name=\'ua-c8f963db-f64a-4acb-8d2d-2a664125061e\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x07\' function=\'0x0\'/>\n </video>\n <memballoon model=\'none\'>\n <alias name=\'balloon0\'/>\n </memballoon>\n <rng model=\'virtio\'>\n <backend model=\'random\'>/dev/urandom</backend>\n <alias name=\'ua-2a267834-ea6b-4a26-94ca-36ab3f22e63a\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x06\' function=\'0x0\'/>\n </rng>\n <panic model=\'pseries\'/>\n <memory model=\'dimm\'>\n <target>\n <size unit=\'KiB\'>262144</size>\n <node>0</node>\n </target>\n <alias name=\'dimm0\'/>\n <address type=\'dimm\' slot=\'0\' base=\'0x40000000\'/>\n </memory>\n <memory model=\'dimm\'>\n <target>\n <size unit=\'KiB\'>786432</size>\n <node>0</node>\n </target>\n <alias name=\'dimm1\'/>\n <address type=\'dimm\' slot=\'1\' base=\'0x50000000\'/>\n </memory>\n <memory model=\'dimm\'>\n <target>\n <size unit=\'KiB\'>1048576</size>\n <node>0</node>\n </target>\n <alias name=\'dimm2\'/>\n <address type=\'dimm\' slot=\'2\' base=\'0x80000000\'/>\n </memory>\n </devices>\n <seclabel type=\'dynamic\' model=\'selinux\' relabel=\'yes\'>\n <label>system_u:system_r:svirt_t:s0:c665,c999</label>\n <imagelabel>system_u:object_r:svirt_image_t:s0:c665,c999</imagelabel>\n </seclabel>\n <seclabel type=\'dynamic\' model=\'dac\' relabel=\'yes\'>\n <label>+107:+107</label>\n <imagelabel>+107:+107</imagelabel>\n </seclabel>\n</domain>\n', 'custom': {}, 'enableGuestEvents': False, 'smpCoresPerSocket': '1', 'vmName': 'memory_hotplug_test', 'guestFQDN': u'dhcp167-10.klab.eng.bos.redhat.com', 'maxMemSize': 16384, 'bootMenuEnable': 'false', 'smpThreadsPerCore': '1', 'memGuaranteedSize': 512, 'arch': 'ppc64le', 'guestIPs': u'10.16.167.10', 'kvmEnable': 'true', 'devices': [{'index': 2, 'iface': 'scsi', 'name': 'sdc', 'vm_custom': {}, 'format': 'raw', 'vmid': 'a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63', 'diskType': 'file', 'alias': 'ua-fb4fbcb3-9589-4aac-9117-bba286811002', 'readonly': 'True', 'specParams': {}, 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '2'}, 'device': 'cdrom', 'discard': False, 'path': '', 'propagateErrors': 'report', 'type': 'disk'}, {'address': {'slot': '0x05', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'reqsize': '0', 'serial': 'c2bd74ad-dfab-416e-b366-42b401de204c', 'index': 0, 'iface': 'virtio', 'apparentsize': '1073741824', 'specParams': {}, 'guestName': '/dev/vda', 'cache': 'none', 'imageID': 'c2bd74ad-dfab-416e-b366-42b401de204c', 'readonly': 'False', 'truesize': '1073741824', 'type': 'disk', 'domainID': '3543271c-284b-461c-be84-8c69b4be7a11', 'volumeInfo': {'path': '/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11', 'type': 'block'}, 'format': 'cow', 'poolID': '74170a2d-821a-402d-8311-8e1cd3cac00a', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11', 'propagateErrors': 'off', 'name': 'vda', 'vm_custom': {}, 'bootOrder': '1', 'vmid': 'a034ccfa-f3ef-4f3b-8945-f26a1fb8ab63', 'volumeID': '393fb11b-f19c-4299-9cd9-cf05b299ee11', 'diskType': 'block', 'alias': 'ua-c2bd74ad-dfab-416e-b366-42b401de204c', 'discard': False, 'volumeChain': [{'domainID': '3543271c-284b-461c-be84-8c69b4be7a11', 'leaseOffset': 112197632, 'volumeID': '393fb11b-f19c-4299-9cd9-cf05b299ee11', 'leasePath': '/dev/3543271c-284b-461c-be84-8c69b4be7a11/leases', 'imageID': 'c2bd74ad-dfab-416e-b366-42b401de204c', 'path': '/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/393fb11b-f19c-4299-9cd9-cf05b299ee11'}, {'domainID': '3543271c-284b-461c-be84-8c69b4be7a11', 'leaseOffset': 109051904, 'volumeID': 'd693d106-ae01-41a5-9e70-562369d838dd', 'leasePath': '/dev/3543271c-284b-461c-be84-8c69b4be7a11/leases', 'imageID': 'c2bd74ad-dfab-416e-b366-42b401de204c', 'path': '/rhev/data-center/mnt/blockSD/3543271c-284b-461c-be84-8c69b4be7a11/images/c2bd74ad-dfab-416e-b366-42b401de204c/d693d106-ae01-41a5-9e70-562369d838dd'}]}], 'status': 'Restoring state', 'maxVCpus': '16', 'clientIp': '', 'statusTime': '43354165420', 'maxMemSlots': 16}} from=::ffff:10.46.16.70,39638, flow_id=87714c7b-a834-4a92-98cc-3e5de8b8b83d, vmId= (api:52)