Description of problem: With kernel-xen-2.6.18-60.el5.ia64, an HP rx2600 w/2GB of RAM in the lab encounters an immediate kernel panic when trying to install a paravirt guest (hardware doesn't support fully virt). Multiple other HP ia64 systems have no problems with this kernel, not sure yet if this is due to the relatively small memory amount on this box + the patch to increase xen default memory allocations or if its something entirely different. 2.6.18-58.el5 is fine, so I'll give a few combinations of builds between there and -60 a go to try to isolate the root cause... Version-Release number of selected component (if applicable): kernel-xen-2.6.18-60.el5 How reproducible: virt-install a pv guest
So... A -52-based kernel + the default memory allocation patch doesn't blow up, but a -59-based kernel + that patch does. First thing Monday will be a -59 build plus the ioremapping fix (otherwise, it won't boot)... Panic dump on the console: (XEN) ia64_fault, vector=0x4, ifa=0xf300000a00003210, iip=0xf000000004076a60, ipsr=0x0000121008226018, isr=0x00000a0400000000 (XEN) Alt DTLB. (XEN) d 0xf000000007bb0080 domid 0 (XEN) vcpu 0xf000000007b90000 vcpu 0 (XEN) (XEN) CPU 0 (XEN) psr : 0000121008226018 ifs : 800000000000040d ip : [<f000000004076a61>] (XEN) ip is at domain_page_flush_and_put+0x441/0x500 (XEN) unat: 0000000000000000 pfs : 000000000000040d rsc : 0000000000000003 (XEN) rnat: 0000000000000000 bsps: f0000000043949a0 pr : 0000000000698999 (XEN) ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f (XEN) csd : 0000000000000000 ssd : 0000000000000000 (XEN) b0 : f000000004076a60 b6 : f0000000040b1e10 b7 : f000000004002e50 (XEN) f6 : 0ffff8000000000000000 f7 : 000000000000000000000 (XEN) f8 : 000000000000000000000 f9 : 000000000000000000000 (XEN) f10 : 000000000000000000000 f11 : 000000000000000000000 (XEN) r1 : f0000000043949a0 r2 : 0000000000000000 r3 : 000000000000003f (XEN) r8 : 0000000000000000 r9 : 0000000000000000 r10 : a000000100a19540 (XEN) r11 : 0000000000000008 r12 : f000000007b97dd0 r13 : f000000007b90000 (XEN) r14 : f000000007b88018 r15 : f300000a00003210 r16 : 0000000000000000 (XEN) r17 : f000000004410480 r18 : 0000000000000000 r19 : 0000000000000001 (XEN) r20 : 0000000000000001 r21 : 0000000000000000 r22 : 0000001008226018 (XEN) r23 : 0000000000000000 r24 : 0000000000000001 r25 : f0000000041a2510 (XEN) r26 : 0000000000000001 r27 : 0000000000000000 r28 : f0000000041a2d10 (XEN) r29 : 0000000000000000 r30 : 0000000000000000 r31 : f0000000041ab180 (XEN) (XEN) Call Trace: (XEN) [<f0000000040b8a30>] show_stack+0x80/0xa0 (XEN) sp=f000000007b97a00 bsp=f000000007b91588 (XEN) [<f00000000407e4e0>] ia64_fault+0xa30/0xad0 (XEN) sp=f000000007b97bd0 bsp=f000000007b91550 (XEN) [<f0000000040b1860>] ia64_leave_kernel+0x0/0x300 (XEN) sp=f000000007b97bd0 bsp=f000000007b91550 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97dd0 bsp=f000000007b914e8 (XEN) [<f00000000407a580>] __dom0vp_add_physmap+0x330/0x630 (XEN) sp=f000000007b97de0 bsp=f000000007b91480 (XEN) [<f00000000405bc70>] do_dom0vp_op+0x1f0/0x560 (XEN) sp=f000000007b97df0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e00 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e00 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e10 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e10 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e20 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e20 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e30 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e30 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e40 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e40 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e50 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e50 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e60 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e60 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e70 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e70 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e80 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e80 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97e90 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97e90 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ea0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ea0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97eb0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97eb0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ec0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ec0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ed0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ed0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ee0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ee0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ef0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ef0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f00 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f00 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f10 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f10 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f20 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f20 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f30 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f30 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f40 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f40 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f50 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f50 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f60 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f60 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f70 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f70 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f80 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f80 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97f90 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97f90 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97fa0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97fa0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97fb0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97fb0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97fc0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97fc0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97fd0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97fd0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97fe0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97fe0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b97ff0 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b97ff0 bsp=f000000007b91440 (XEN) [<f000000004002e80>] fast_hypercall+0x170/0x2f0 (XEN) sp=f000000007b98000 bsp=f000000007b91440 (XEN) [<f000000004076a60>] domain_page_flush_and_put+0x440/0x500 (XEN) sp=f000000007b98000 bsp=f000000007b91440 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) Fault in Xen. (XEN) **************************************** (XEN) (XEN) Reboot in five seconds...
I ran a test of rebooting a pv guest repeatedly over the weekend on another rx2600 with no panic seen. It appears that this issue is specific to hp-lp1. I will look at the differences between the two boxes.
Took a few days to get ahold of the machine again, someone else had it reserved. Everything works fine w/the -59 + ioremap fix build I threw together, so the problem is definitely something that went into -60.
So... To make things even more interesting, I'm running a -61-based kernel-xen on the same previously crashy rx2600, and its working just fine now... Will roll back to -60 again in a moment to see if that still crashes. If so, it seems something in -61 fixed a problem in -60, and we can just close this bug out. Though it'd be nice to know exactly what the regression was, since it was only seen on this one box, its probably not worth the effort to track down.
Figures. Now I can't even get the problem to reproduce on the -60 kernel. Ugh. Just gonna close this CANTFIX for now since its only been seen briefly on this one machine, and we can reopen if the problem shows up again. :\