Description of problem:
I was testing out FV save/restore functionality on RHEL-5.3, with dom0 x86_64 kernel 2.6.18-125.el5, xen-3.0.3-73, and FV x86_64 kernel 2.6.18-125.el5.  My FV guest is assigned 8 vcpus, and no PV-on-HVM devices (this is important).  I successfully boot up my guest, and then execute:

# xm save rhel5fv_x86_64 /var/lib/xen/save/rhel5fv_x86_64-save

That appears to successfully complete.  Then I do:

# xm restore /var/lib/xen/save/rhel5fv_x86_64-save

Which also appears to successfully complete, except that all 8 processors are just spinning, and the guest never returns.

I took a core-dump of the stuck domain, and saw this:

PID: 3003   TASK: ffff81002fea5040  CPU: 0   COMMAND: "suspend"
 #0 [ffff81002aa49ca0] schedule at ffffffff80063035
 #1 [ffff81002aa49ca8] thread_return at ffffffff80063097
 #2 [ffff81002aa49d68] __next_cpu at ffffffff80148b8b
 #3 [ffff81002aa49db8] physflat_send_IPI_allbutself at ffffffff80079441
 #4 [ffff81002aa49e18] __smp_call_function at ffffffff800759f6
 #5 [ffff81002aa49e68] smp_call_function at ffffffff80075b2c
 #6 [ffff81002aa49e98] __xen_suspend at ffffffff881b69a1
 #7 [ffff81002aa49ed8] xen_suspend at ffffffff881b6615
 #8 [ffff81002aa49ee8] kthread at ffffffff800324b3
 #9 [ffff81002aa49f48] kernel_thread at ffffffff8005dfb1

PID: 0      TASK: ffff81002fc34100  CPU: 1   COMMAND: "swapper"
 #0 [ffff81002fc55e18] schedule at ffffffff80063035
 #1 [ffff81002fc55e20] thread_return at ffffffff80063097
 #2 [ffff81002fc55ec8] default_idle at ffffffff8006b2b3
 #3 [ffff81002fc55ef0] cpu_idle at ffffffff80048e69

(repeat CPU 1 entry for the other 6 CPUs)

So, here's what happened:
1)  The xm save command caused a "suspend" value to be written to xenbus.
2)  This, in turn, fired a watch in the guest kernel.  The guest kernel responds by starting to run the xen_suspend() function, which does an "smp_call_function" to quiesce all of the other CPUs.
3)  However, before this action could complete inside the guest, the tools on the dom0 decided that this was an HVM domain, and further, *it doesn't know that PV-on-HVM drivers are active*.  Because of this, the tools on the dom0 just pull the plug, save the memory, and kill the domain.
4)  Now, on resume, everything is put back in place, but the guest kernel still thinks it is shutting down.  At this point, we are completely wedged.

In the tools, here's what's happening:
    def shutdown(self, reason):
        if not reason in shutdown_reasons.values():
            raise XendError('Invalid reason: %s' % reason)
        if self.domid == 0:
            raise XendError("Can't specify Domain-0")
        self.storeDom("control/shutdown", reason)

        # HVM domain shuts itself down only if it has PV drivers
        if self.is_hvm():
            hvm_pvdrv = xc.hvm_get_param(self.domid, HVM_PARAM_CALLBACK_IRQ)
            if not hvm_pvdrv:
                code = reverse_shutdown_reasons[reason]
                xc.domain_destroy_hook(self.domid)
                log.info("HVM save:remote shutdown dom %d!", self.domid)
                xc.domain_shutdown(self.domid, code)

As you can see, it queries the HVM_PARAM_CALLBACK_IRQ to see if it has a value.  This value is stored in the hypervisor.  However, for some reason (which I haven't yet determined), this comes back as false, so the tools go on to pull the plug on the domain without waiting for the domain to shut itself down.

Note that I believe the RHEL-4 kernel has the same problem, although I have not yet verified that this is the case.