Bug 475849 - [RHEL 4.7 Xen]: Guest hang on FV save/restore
[RHEL 4.7 Xen]: Guest hang on FV save/restore
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel-xen (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Chris Lalancette
Martin Jenner
: 456626 (view as bug list)
Depends On: 475778
  Show dependency treegraph
Reported: 2008-12-10 14:24 EST by Don Dutile
Modified: 2009-05-18 15:28 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-05-18 15:28:08 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Patch to fix the FV save/restore issues (2.68 KB, patch)
2008-12-17 10:29 EST, Chris Lalancette
no flags Details | Diff

  None (edit)
Description Don Dutile 2008-12-10 14:24:12 EST
+++ This bug was initially created as a clone of Bug #475778 +++

Description of problem:
I was testing out FV save/restore functionality on RHEL-5.3, with dom0 x86_64 kernel 2.6.18-125.el5, xen-3.0.3-73, and FV x86_64 kernel 2.6.18-125.el5.  My FV guest is assigned 8 vcpus, and no PV-on-HVM devices (this is important).  I successfully boot up my guest, and then execute:

# xm save rhel5fv_x86_64 /var/lib/xen/save/rhel5fv_x86_64-save

That appears to successfully complete.  Then I do:

# xm restore /var/lib/xen/save/rhel5fv_x86_64-save

Which also appears to successfully complete, except that all 8 processors are just spinning, and the guest never returns.

I took a core-dump of the stuck domain, and saw this:

PID: 3003   TASK: ffff81002fea5040  CPU: 0   COMMAND: "suspend"
 #0 [ffff81002aa49ca0] schedule at ffffffff80063035
 #1 [ffff81002aa49ca8] thread_return at ffffffff80063097
 #2 [ffff81002aa49d68] __next_cpu at ffffffff80148b8b
 #3 [ffff81002aa49db8] physflat_send_IPI_allbutself at ffffffff80079441
 #4 [ffff81002aa49e18] __smp_call_function at ffffffff800759f6
 #5 [ffff81002aa49e68] smp_call_function at ffffffff80075b2c
 #6 [ffff81002aa49e98] __xen_suspend at ffffffff881b69a1
 #7 [ffff81002aa49ed8] xen_suspend at ffffffff881b6615
 #8 [ffff81002aa49ee8] kthread at ffffffff800324b3
 #9 [ffff81002aa49f48] kernel_thread at ffffffff8005dfb1

PID: 0      TASK: ffff81002fc34100  CPU: 1   COMMAND: "swapper"
 #0 [ffff81002fc55e18] schedule at ffffffff80063035
 #1 [ffff81002fc55e20] thread_return at ffffffff80063097
 #2 [ffff81002fc55ec8] default_idle at ffffffff8006b2b3
 #3 [ffff81002fc55ef0] cpu_idle at ffffffff80048e69

(repeat CPU 1 entry for the other 6 CPUs)

So, here's what happened:
1)  The xm save command caused a "suspend" value to be written to xenbus.
2)  This, in turn, fired a watch in the guest kernel.  The guest kernel responds by starting to run the xen_suspend() function, which does an "smp_call_function" to quiesce all of the other CPUs.
3)  However, before this action could complete inside the guest, the tools on the dom0 decided that this was an HVM domain, and further, *it doesn't know that PV-on-HVM drivers are active*.  Because of this, the tools on the dom0 just pull the plug, save the memory, and kill the domain.
4)  Now, on resume, everything is put back in place, but the guest kernel still thinks it is shutting down.  At this point, we are completely wedged.

In the tools, here's what's happening:
    def shutdown(self, reason):
        if not reason in shutdown_reasons.values():
            raise XendError('Invalid reason: %s' % reason)
        if self.domid == 0:
            raise XendError("Can't specify Domain-0")
        self.storeDom("control/shutdown", reason)

        # HVM domain shuts itself down only if it has PV drivers
        if self.is_hvm():
            hvm_pvdrv = xc.hvm_get_param(self.domid, HVM_PARAM_CALLBACK_IRQ)
            if not hvm_pvdrv:
                code = reverse_shutdown_reasons[reason]
                log.info("HVM save:remote shutdown dom %d!", self.domid)
                xc.domain_shutdown(self.domid, code)

As you can see, it queries the HVM_PARAM_CALLBACK_IRQ to see if it has a value.  This value is stored in the hypervisor.  However, for some reason (which I haven't yet determined), this comes back as false, so the tools go on to pull the plug on the domain without waiting for the domain to shut itself down.

Note that I believe the RHEL-4 kernel has the same problem, although I have not yet verified that this is the case.

--- Additional comment from ddutile@redhat.com on 2008-12-10 14:20:52 EDT ---

Created an attachment (id=326538)
Proposed patch; seems to be working on test case (save/restore w/parallel make in background)

From upstream xen-unstable, cset 18669
Comment 1 Don Dutile 2008-12-10 14:26:20 EST
Same patch that applied to rhel5.3(5.4?) applies as-is for rhel4.
Comment 2 Chris Lalancette 2008-12-12 07:26:42 EST
I tried this patch on RHEL-4, but no dice.  I still completely hang on save, in this case.  Actually, I tried out a whole slew of kernels going all the way back to -67.EL.  Things worked up until 2.6.9-68.32, which happens to be the development version where the the PV-on-HVM drivers were added to the RHEL-4 bare-metal kernel.  So, I would say that this functionality has probably been broken since then.  I do not know about the xenpv async drivers; I have not tried them.

Chris Lalancette
Comment 3 Chris Lalancette 2008-12-17 08:57:04 EST
OK.  I found the problem.  When we initially did the RHEL-4 Xen backport, we put some conditional code in kernel/kthread.c to be ultra-careful, and not change bare-metal.  However, it turns out that this is too careful; upstream Linux long ago went the way our PV guests are doing things, and from the looks of it our FV guests also need to do the same.  For reference, the upstream hash that changed the code in kernel/kthread.c is 3ba06378990208fce98e95d52fb4d0c3ff8b3c09; it's in the old-2.6-bkcvs tree, which pre-dates 2.6.12.  With that patch in place, I now get to the livelock from the RHEL-5 bug.  I'm going to test out these two patches together and report back results.

Chris Lalancette
Comment 4 Chris Lalancette 2008-12-17 10:29:35 EST
Created attachment 327261 [details]
Patch to fix the FV save/restore issues
Comment 5 Vivek Goyal 2009-01-07 09:37:43 EST
Committed in 78.24.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Comment 7 Chris Lalancette 2009-02-24 09:11:53 EST
*** Bug 456626 has been marked as a duplicate of this bug. ***
Comment 10 errata-xmlrpc 2009-05-18 15:28:08 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.