Bisecting of upstream kernel shows following offending commit:

commit 448f2831934381e9d3c4d93e700ba7bbe14612dc
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge>
Date:   Wed Dec 15 13:19:33 2010 +0000

    xen: add m2p override mechanism

(In reply to comment #1)
> Bisecting of upstream kernel shows following offending commit:
> 
> commit 448f2831934381e9d3c4d93e700ba7bbe14612dc
> Author: Jeremy Fitzhardinge <jeremy.fitzhardinge>
> Date:   Wed Dec 15 13:19:33 2010 +0000
> 
>     xen: add m2p override mechanism

Bug reproduces only on more than 1 numa nodes machines,
tested it on 2 sockets intel/amd boxes.

Guset can be restored successfully by turning off 'numa' option on xen hv commandline. 

In addition tested it with SLES 11 host (xen 4.0), it also works as expected
with and without 'numa=on' option.

Looks like xen 3.0 bug.

Based on comment 3 this sounds like a RHEL5 Xen HV problem. Moving it there.

Just retested with latest RHEL5.8 beta xen and FC16 (3.1.5-2.fc16.i686.PAE).

(XEN) mm.c:630:d2 Non-privileged (2) attempt to map I/O space 003c092a
(XEN) mm.c:3624:d2 ptwr_emulate: could not get_page_from_l1e()
(XEN) Pagetable walk from 00000000ebc0eff4:
(XEN)  L4[0x000] = 0000000087ef9027 00000000000018b9
(XEN)  L3[0x003] = 0000000087bff027 0000000000001fb7
(XEN)  L2[0x15e] = 000000008a49b067 0000000000003bd2 
(XEN)  L1[0x00e] = 800000030ecb0061 000000000002bc0e
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 2 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.1.2-300.el5  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    0061:[<00000000c092a346>]
(XEN) RFLAGS: 0000000000010217   CONTEXT: guest
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 000000000000007b
(XEN) rdx: 00000000ebc63ed0   rsi: 00000000ebc0f078   rdi: 00000000c092a310
(XEN) rbp: 00000000ebc0f070   rsp: 00000000ebc0effc   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000030f364000   cr2: 00000000ebc0eff8
(XEN) ds: 007b   es: 007b   fs: 00d8   gs: 00e0   ss: 0069   cs: 0061
(XEN) Guest stack trace from esp=ebc0effc:
(XEN)   80000003 00000007 00000001 ebc63ed0 ebc0d010 00000000 00000000 00000028
(XEN)   00000001 ebc0d030 ebc0d038 ebc030e0 ebc00300 00000003 00000001 ebc0d048
(XEN)   ebc0d050 ebc0c0c0 ebc00330 00000000 00000000 00000000 00000000 00000000
(XEN)   00000000 f0000000 00000000 ebc0f138 c092a310 ebc0f0b4 c0927997 00000000
(XEN)   0000007b ebc63ed0 ebc0f138 c092a310 ebc0f0b4 f5298000 0000007b 0000007b
(XEN)   000000d8 000000e0 ffffffff c04053ce 00010061 00010017 ebc0f130 c092a34b
(XEN)   00000000 f0000001 0001a1ff ebc63ed0 00000000 00000000 00000000 00000028
(XEN)   00000000 ebc0f060 c0b9e340 00000000 ebc003c0 00000000 00000000 00000001
(XEN)   00000000 00000000 ebc0f104 ebc0f104 00000000 74656e03 00000000 00000000
(XEN)   00000000 f0000002 00000000 ebc0f1f8 c092a310 ebc0f174 c0927997 00000000
(XEN)   0000007b ebc63ed0 ebc0f1f8 c092a310 ebc0f174 f5298000 0000007b 0000007b
(XEN)   000000d8 000000e0 ffffffff c04053ce 00010061 00010017 ebc0f1f0 c092a34b
(XEN)   00000000 f0000003 0005416d ebc63ed0 00000000 00000000 00000000 00000028
(XEN)   c0945280 ebc0f120 c0b9e340 ebdb5240 00000000 00000000 00000000 00000001
(XEN)   00000000 00000000 ebc0f1c4 ebc0f1c4 00000000 00736602 00000000 00000000
(XEN)   00000000 f0000004 00000000 ebc0f2b8 c092a310 ebc0f234 c0927997 00000000
(XEN)   0000007b ebc63ed0 ebc0f2b8 c092a310 ebc0f234 f5298000 0000007b 0000007b
(XEN)   000000d8 000000e0 ffffffff c04053ce 00010061 00010017 ebc0f2b0 c092a34b
(XEN)   00000000 f0000005 0002416d ebc63ed0 00000000 00000000 00000000 00000028
(XEN)   c0945280 00000000 ebc0f180 00000000 00000000 00000000 00000000 00000001

Created attachment 549882 [details]
[RHEL5.9 Xen PATCH] x86: fix a few 32-on-64 compat mode issues

Bug uncovered by commit 448f28319343 in upstream kernel by introducing
a following call chain on restore:

      xen_arch_post_suspend
        xen_setup_shared_info
          set_fixmap
            xen_set_fixmap
              mfn_pte
                mfn_to_pfn
                  m2p_find_override
                    spin_lock_irqsave
                      ...
                        xen_save_fl
                          vcpu = percpu_read(xen_vcpu);
                          flags = !vcpu->evtchn_upcall_mask;

where guest tries to access invalid vcpu_info memory area. Since
VCPUOP_register_vcpu_info isn't properly handled by xen for 32-on-64 case
and as result mapping for that memory area isn't saved in restore file.
Consequently on restore xen_vcpu points to a not mapped memory.

xen-unstable c/s 17612 fixes issue by properly handling VCPUOP_register_vcpu_info for 32-on-64bit case.

Probably guest was lucky on amd box with numa=off. Since I was able to reproduce BUG on intel systems with several cpus without numa=on.

What about 5.8 or at least 5.8.z?  This is in some sense a regression.

why is it regression, it was there forever.

It shows as a regression when you upgrade the guest; migration no longer works.

I agree with 5.8.z, but it's not a high enough priority issue to slip 5.8's schedule.

Patch(es) available in kernel-2.6.18-305.el5
You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5/
Detailed testing feedback is always welcomed.
If you require guidance regarding testing, please ask the bug assignee.

(In reply to comment #7)
> Probably guest was lucky on amd box with numa=off. Since I was able to
> reproduce BUG on intel systems with several cpus without numa=on.

On what intel system did you reproduce it without numa on please?

(In reply to comment #15)
> (In reply to comment #7)
> > Probably guest was lucky on amd box with numa=off. Since I was able to
> > reproduce BUG on intel systems with several cpus without numa=on.
> 
> On what intel system did you reproduce it without numa on please?

on my ws (it's dell-pet610 with 2 sockets) and the other one I do not remember.

I reproduced this on an Intel(E5620) machine with 2 sockets, with the RHEL 5.7 release kernel-xen (-274).

The guest is Fedora-16 32 bit PV with kernel-PAE-3.1.0-7.fc16.i686.

With kernel-xen-274, either with numa=on or off, the guest crashes upon restore. `xm dmesg` shows the crash message:
Domain 2 (vcpu#0) crashed on cpu#15:
...

With kernel-xen-305, the guest does not crash on restore, either with numa on or off.

Also tried RHEL5 and RHEL6 32bit PV guests. This bug does not apply to them.

Thanks for verifying, Jinxin! Can you please move the bug to VERIFIED?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0150.html