Bug 513431

Hmm, I can't reproduce it... Could you provide xend.log from the target machine? When migrating, xend.log from the source machine is rarely useful.

Thanks

I can reproduce this.  After migration, the guest is indeed in a "confused" state:

Name                                      ID Mem(MiB) VCPUs State   Time(s)
RHEL5-64-HVM                               2      511     1 ------      0.4

Created attachment 355000 [details]
xend.log file from *source* machine

I stand corrected, I cannot reproduce this.  The target machine didn't have VT enabled.  The xend.log file there is quite clear:

[2009-07-24 13:17:26 xend 5739] ERROR (XendDomain:285) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 280, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, relocating=relocating)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 166, in restore
    dominfo = xd.restore_(vmconfig, relocating)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 305, in restore_
    dominfo = XendDomainInfo.restore(config, relocating)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 292, in restore
    vm.construct()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1787, in construct
    raise VmError("HVM guest support is unavailable: is VT/AMD-V "
VmError: HVM guest support is unavailable: is VT/AMD-V supported by your CPU and enabled in your BIOS?

Maybe something similar is happening for you?

I'm creating another bug for the "confusion upon failed migration".

(In reply to comment #5)
>     raise VmError("HVM guest support is unavailable: is VT/AMD-V "
> VmError: HVM guest support is unavailable: is VT/AMD-V supported by your CPU
> and enabled in your BIOS?
> 
> Maybe something similar is happening for you?  

My tests were run over rhts which was scheduled to run in machines that were hvm enabled. However, i'll go in and check to ensure and also to get xend.log from the destination machine and will update the BZ.

(In reply to comment #7)
> (In reply to comment #5)
> >     raise VmError("HVM guest support is unavailable: is VT/AMD-V "
> > VmError: HVM guest support is unavailable: is VT/AMD-V supported by your CPU
> > and enabled in your BIOS?
> > 
> > Maybe something similar is happening for you?  
> 
> My tests were run over rhts which was scheduled to run in machines that were
> hvm enabled. However, i'll go in and check to ensure and also to get xend.log
> from the destination machine and will update the BZ.  

Oy.. Looks like the setup was wrong, both sides had HVM machines however during installation the destination didn't get the shared drive with image files set up correctly. 
  What has really confused me was that the paravirt migrations still do happen, even though there are no images of them on the destination machine either. How could this happen? Shouldn't it also error like hvm guests happen?
  Upon closer inspection, the paravirt guests aren't usable after the migration, you can connect to console but can't do much, nonetheless the migration still happens. Should this be an issue? Paolo , what's the number of the other bug you've created?

Jiri told me he was already aware of that bug, it is https://bugzilla.redhat.com/show_bug.cgi?id=486308

Gurhan, are you really sure this is a regression (since you changed the summary)?

Yeah, since this is a completely different bug which has been with us for-probably-ever, we should remove Regression keyword... Any objections, Gurhan?

(In reply to comment #12)
> Yeah, since this is a completely different bug which has been with us
> for-probably-ever, we should remove Regression keyword... Any objections,
> Gurhan?  

Nope not at all, sorry i forgot to remove the keyword when i made changes to the BZ.

So, does this affect PV or HVM guests? There's something about HVM guests but title states something about PV guest migration.

Thanks,
Michal

Well, I was able to reproduce it so this is still the issue. Unfortunately situation here is not that easy since the guest memory gets transferred to the target machine but then it fails with reason that the device could not be connected:

[2010-06-09 13:12:57 xend 4400] ERROR (XendCheckpoint:279) Device 51728 (vbd) could not be connected. /home2/test.img does not exist.
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 277, in restore
    dominfo.waitForDevices() # Wait for backends to set up
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2423, in waitForDevices
    self.waitForDevices_(c)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1442, in waitForDevices_
    return self.getDeviceController(deviceClass).waitForDevices()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py", line 162, in waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/DevController.py", line 183, in waitForDevice
    raise VmError("Device %s (%s) could not be connected. "
VmError: Device 51728 (vbd) could not be connected. /home2/test.img does not exist.

The guest was working for me since I've been having using it as xvdb device but the device was not present in the guest so when having the boot disk inaccessible it won't be working at all. However, since the data are transferred in the beginning of the connection (before the guest memory) it should be checked here and an error could be returned before the memory is being transferred at all resulting into guest not being migrated and to resume on the source machine. I'll try to investigate this further.

Michal

Created attachment 422529 [details]
Add check for disk image existence on restore/migrate

Well, investigation revealed that there's a very good option to fix it since prior to the guest's memory gets transferred there is a VM configuration transfer so I just defined a function to parse the disk unames for vbd and tap devices and check for the existence. If the file doesn't exists an error is returned as can be seen on following lines:

[2010-06-09 14:22:20 xend 16856] ERROR (XendDomain:284) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 279, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, relocating=relocating)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 182, in restore
    check_file_exists(vmconfig)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 160, in check_file_exists
    raise XendError("Disk image %s doesn't exist" % fn)
XendError: Disk image /home2/test.img doesn't exist

It's been tested with both PV and HVM guests and it was working fine.

Upstream doesn't seem to have this fixed but unfortunately I'm now having 2 machines with working upstream Xen so I need to install second machine with upstream Xen-4.1 to see whether it's an issue or not.

Michal

+        if (type(x) != str):
+            if x[0] == 'device':
+                if x[1][0] in ('tap', 'vbd'):

 if type(x) != str and x[0] == 'device' and x[1][0] in ('tap', 'vbd'):

?

or maybe

 if type(x) == list and x[0] == 'device' and x[1][0] in ('tap', 'vbd'):

+                        if (type(p) != str):
+                            if p[0] == 'uname':

Likewise,

 if type(p) == list and p[0] == 'uname':

would help saving some indentation.  Otherwise looks fine!

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0031.html