Red Hat Bugzilla – Bug 475849
[RHEL 4.7 Xen]: Guest hang on FV save/restore
Last modified: 2009-05-18 15:28:08 EDT
+++ This bug was initially created as a clone of Bug #475778 +++
Description of problem:
I was testing out FV save/restore functionality on RHEL-5.3, with dom0 x86_64 kernel 2.6.18-125.el5, xen-3.0.3-73, and FV x86_64 kernel 2.6.18-125.el5. My FV guest is assigned 8 vcpus, and no PV-on-HVM devices (this is important). I successfully boot up my guest, and then execute:
# xm save rhel5fv_x86_64 /var/lib/xen/save/rhel5fv_x86_64-save
That appears to successfully complete. Then I do:
# xm restore /var/lib/xen/save/rhel5fv_x86_64-save
Which also appears to successfully complete, except that all 8 processors are just spinning, and the guest never returns.
I took a core-dump of the stuck domain, and saw this:
PID: 3003 TASK: ffff81002fea5040 CPU: 0 COMMAND: "suspend"
#0 [ffff81002aa49ca0] schedule at ffffffff80063035
#1 [ffff81002aa49ca8] thread_return at ffffffff80063097
#2 [ffff81002aa49d68] __next_cpu at ffffffff80148b8b
#3 [ffff81002aa49db8] physflat_send_IPI_allbutself at ffffffff80079441
#4 [ffff81002aa49e18] __smp_call_function at ffffffff800759f6
#5 [ffff81002aa49e68] smp_call_function at ffffffff80075b2c
#6 [ffff81002aa49e98] __xen_suspend at ffffffff881b69a1
#7 [ffff81002aa49ed8] xen_suspend at ffffffff881b6615
#8 [ffff81002aa49ee8] kthread at ffffffff800324b3
#9 [ffff81002aa49f48] kernel_thread at ffffffff8005dfb1
PID: 0 TASK: ffff81002fc34100 CPU: 1 COMMAND: "swapper"
#0 [ffff81002fc55e18] schedule at ffffffff80063035
#1 [ffff81002fc55e20] thread_return at ffffffff80063097
#2 [ffff81002fc55ec8] default_idle at ffffffff8006b2b3
#3 [ffff81002fc55ef0] cpu_idle at ffffffff80048e69
(repeat CPU 1 entry for the other 6 CPUs)
So, here's what happened:
1) The xm save command caused a "suspend" value to be written to xenbus.
2) This, in turn, fired a watch in the guest kernel. The guest kernel responds by starting to run the xen_suspend() function, which does an "smp_call_function" to quiesce all of the other CPUs.
3) However, before this action could complete inside the guest, the tools on the dom0 decided that this was an HVM domain, and further, *it doesn't know that PV-on-HVM drivers are active*. Because of this, the tools on the dom0 just pull the plug, save the memory, and kill the domain.
4) Now, on resume, everything is put back in place, but the guest kernel still thinks it is shutting down. At this point, we are completely wedged.
In the tools, here's what's happening:
def shutdown(self, reason):
if not reason in shutdown_reasons.values():
raise XendError('Invalid reason: %s' % reason)
if self.domid == 0:
raise XendError("Can't specify Domain-0")
# HVM domain shuts itself down only if it has PV drivers
hvm_pvdrv = xc.hvm_get_param(self.domid, HVM_PARAM_CALLBACK_IRQ)
if not hvm_pvdrv:
code = reverse_shutdown_reasons[reason]
log.info("HVM save:remote shutdown dom %d!", self.domid)
As you can see, it queries the HVM_PARAM_CALLBACK_IRQ to see if it has a value. This value is stored in the hypervisor. However, for some reason (which I haven't yet determined), this comes back as false, so the tools go on to pull the plug on the domain without waiting for the domain to shut itself down.
Note that I believe the RHEL-4 kernel has the same problem, although I have not yet verified that this is the case.
--- Additional comment from email@example.com on 2008-12-10 14:20:52 EDT ---
Created an attachment (id=326538)
Proposed patch; seems to be working on test case (save/restore w/parallel make in background)
From upstream xen-unstable, cset 18669
Same patch that applied to rhel5.3(5.4?) applies as-is for rhel4.
I tried this patch on RHEL-4, but no dice. I still completely hang on save, in this case. Actually, I tried out a whole slew of kernels going all the way back to -67.EL. Things worked up until 2.6.9-68.32, which happens to be the development version where the the PV-on-HVM drivers were added to the RHEL-4 bare-metal kernel. So, I would say that this functionality has probably been broken since then. I do not know about the xenpv async drivers; I have not tried them.
OK. I found the problem. When we initially did the RHEL-4 Xen backport, we put some conditional code in kernel/kthread.c to be ultra-careful, and not change bare-metal. However, it turns out that this is too careful; upstream Linux long ago went the way our PV guests are doing things, and from the looks of it our FV guests also need to do the same. For reference, the upstream hash that changed the code in kernel/kthread.c is 3ba06378990208fce98e95d52fb4d0c3ff8b3c09; it's in the old-2.6-bkcvs tree, which pre-dates 2.6.12. With that patch in place, I now get to the livelock from the RHEL-5 bug. I'm going to test out these two patches together and report back results.
Created attachment 327261 [details]
Patch to fix the FV save/restore issues
Committed in 78.24.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
*** Bug 456626 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.