Bug 624598 - Win7 and Windows 2008 R2 xen guests with multiple vcpus can't restart
Win7 and Windows 2008 R2 xen guests with multiple vcpus can't restart
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel-xen (Show other bugs)
5.6
All Linux
high Severity high
: rc
: ---
Assigned To: Paolo Bonzini
Virtualization Bugs
: TestOnly
: 574123 636421 657281 (view as bug list)
Depends On: 625902 625903
Blocks: 526415
  Show dependency treegraph
 
Reported: 2010-08-17 03:33 EDT by shaochen
Modified: 2011-01-13 16:09 EST (History)
13 users (show)

See Also:
Fixed In Version: kernel-2.6.18-222.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 625902 625903 (view as bug list)
Environment:
Last Closed: 2011-01-13 16:09:57 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
xm dmesg (15.09 KB, text/plain)
2010-08-17 03:36 EDT, shaochen
no flags Details
xend.log (85.88 KB, text/plain)
2010-08-17 04:14 EDT, shaochen
no flags Details
Windows event log XML file output for Kernel Power Manager (7.67 KB, application/xml)
2010-08-17 09:00 EDT, Michal Novotny
no flags Details
add new xend.log (122.75 KB, text/plain)
2010-08-18 03:31 EDT, shaochen
no flags Details

  None (edit)
Description shaochen 2010-08-17 03:33:54 EDT
Description of problem:
Set the Vcpu numbers >=2, the Windows 2008 R2 guest and Win7 guest can't restart, the guest will hung on shutting down screen.

Version-Release number of selected component (if applicable):
xen-3.0.3-115.el5

How reproducible:
100%

Steps to Reproduce:
1.Install windows2008R2 (or Win7) guest in Rhel5.
2.Set vcpu >=2 in xml file.
3.Reboot guest.
  
Actual results:
The Windows 2008 R2 guest and Win7 guest can't restart, the guest will hung on shutting down screen.

Expected results:
Windows should be restart smoothly.

Additional info:
CPU model: Intel
The same issue happens when we set smp form 2 to 16(maximum virtual CPUs). 
When we set vcpu=1, the windows 2008 r2 and Win7 can restart smoothly.
Comment 1 shaochen 2010-08-17 03:36:36 EDT
Created attachment 439066 [details]
xm dmesg
Comment 2 shaochen 2010-08-17 03:39:27 EDT
Set the Priority to high, because this bug block WHQL test in win2008R2 and Win7.
Comment 3 shaochen 2010-08-17 04:14:38 EDT
Created attachment 439068 [details]
xend.log
Comment 4 Miroslav Rezanina 2010-08-17 04:28:48 EDT
Windows guest can't be rebooted when vcpus > 1 is used.
Comment 5 Michal Novotny 2010-08-17 07:13:57 EDT
I can't see anything relevant in xend.log and attached version of `xm dmesg` output but when I tried config of Windows 2008 R2 x64 I've seen some other messages at the end of `xm dmesg` as well:

(XEN) vlapic.c:293:d2 Ignoring guest NMI
(XEN) vlapic.c:293:d2 Ignoring guest NMI

This is there only twice and the context is:

#/usr/lib64/xen/bin/xenctx 2
rip: fffff800014191bd 
rsp: fffff88001f40ba0
rax: 00000003   rbx: fffff80001638e80   rcx: f803ed0f63590000   rdx: 8fa9fbff
rsi: 00000001   rdi: 00000000   rbp: 00000001
 r8: fffff88001f40b20    r9: fffff88001f40b78   r10: 00000008   r11: 0001067a
r12: fffff8000166de00   r13: 00000000   r14: 00000000   r15: 00000001
 cs: 00000010    ds: 00000000    fs: 00000000    gs: 00000000

For 4 VCPUs the contexts seems to be the same but there were 3 lines about ignoring guest NMI from this guest shutdown attempt:

(XEN) vlapic.c:293:d3 Ignoring guest NMI
(XEN) vlapic.c:293:d3 Ignoring guest NMI
(XEN) vlapic.c:293:d3 Ignoring guest NMI

When I destroyed the guest and set VCPU count to 1 again the guest was able to shutdown successfully.

There's a bug 502543 which is about the same guest running on the top of KVM virtualization and this seems to be the issue in svm.c/vmx.c implementation.

Quick analysis of hypervisor coding says that this is coming from arch/x86/hvm/vlapic.c file of vlapic_accept_irq() function and those are being printed only for VCPU 2 and any further (not VCPU 1). I guess ignoring those non-maskable interrupts are the key to solve the problem. Since this may be in VMX/SVM implementation itself I did try testing on both Intel and AMD machines to determine whether it's problem in the VMX only implementation and I on AMD machine the guest was able to shutdown (after a short delay but on Intel it was stuck indefinitely - also with no "Ignoring guest NMI" message) with 2 VCPUs so I guess the issue is definitely in the VMX implementation.

In the vlapic.c file there a code like:

        /* FIXME How to check the situation after vcpu reset? */
        if ( v->is_initialised )
            hvm_vcpu_reset(v);

in the arch/x86/hvm/vlapic.c file for APIC_DM_INIT but I don't know whether this could be relevant. Nevertheless I found it strange that one (first) VCPU can process the NMI but not the others. Like I say, I think the NMI is the key to solve this one. I did try to find the relevant codeset in the upstream but I was unable to find any. I'm going to test this issue on the upstream version and we'll see if this is the issue with the upstream version.

Michal
Comment 6 Michal Novotny 2010-08-17 08:40:37 EDT
Well, I've tested this on upstream Xen-4.1 running on kernel 2.6.32.15 (pvops) and the guest was able to shutdown successfully when using 2 VCPUs so no issue upstream. There were no messages about ignored guest NMIs. Unfortunately I don't know about relevant upstream changesets, I've just found something about MCE/NMI injection on [1] but I don't know whether it could be relevant.

I did also some more testing with the RHEL version and I tried running xenctx on the domain after shutdown command, all the registers remained the same for about 2 minutes but the RIP register was changing for about 20 seconds and the value of RIP register, i.e. the instruction pointer, was fffff800019f31bf for about minute and half, then it started changing between fffff800019f31c5, fffff800019f31c8 and fffff800019f31bf for about 10 seconds and then it stopped changing for about 30 seconds. Considering all I know so far I think that the guest is trying to alter state of NMI for those VCPUs and it won't give up. Since those NMIs are being ignored it's being stuck indefinitely.

Since this happens only with VMX implementation (i.e. Intel only) and that there's a VMXAssist code in the user-space stack, I kept that in mind when working on this one but from what I know VMXAssist is being used only as the part of hvmloader, i.e. for guest boot up so basically I think this is the hypervisor related issue.

Also, considering the fact some instructions are being processed in the guest we can add some instruction debugging into the qemu-dm on the user-space stack to see what instructions are being called there - I think those instructions are somehow related to the ignored NMIs and since it's being ignored but the guest is trying to access it and waiting for reply, the guest ends up stuck.

Michal

[1] http://xenbits.xensource.com/staging/xen-unstable.hg?rev/14d9fb7a3262
Comment 7 Michal Novotny 2010-08-17 09:00:53 EDT
Created attachment 439103 [details]
Windows event log XML file output for Kernel Power Manager

I tried to have a look to the Event Log Viewer in the guest (attached) and I found out something relevant that came to my mind when studying the Event Logs.

There is a notification of system shutdown being initiated but after that there's a registration of all the CPUs for the Windows Kernel Power provider, more specifically Microsoft-Windows-Kernel-Processor-Power for processor power. This is being registered on the Windows start up but when you just shutdown the guest and edit the domain configuration file to add some VCPU the notification is not currently present in the system and it appears that it's not trying to shutdown the new VCPU and therefore it's being stuck.

As a proof of my theory I destroyed the guest that was on stuck in the "Shutting down" state and started it up again (to take the XML file from the Event Log). After transferring the XML file I tried to shutdown the guest using normal Windows way and although it took some time it passed and the guest was able to shutdown successfully with no "Ignoring guest NMI" message in the `xm dmesg` output so I'm thinking what would it do on a bare metal system when adding a new CPU but unfortunately I can't test it.

The steps to be done on a bare-metal system should be:
  [ Use system with 2 processor sockets but only one occupied with a CPU ]
1) Install Windows 2008 R2 x64 - the one I test with too
2) Shutdown the guest when fully installed and install a new CPU to socket 2
3) Run the Windows system
4) The system should detect a new CPU and require restart
5) Click "Restart now"

If the system is stuck and you need to restart the machine you're running into the same issues that this bugzilla is about so it's basically an issue with the Windows system itself. If you don't run into those issues, then this is software (hypervisor) related.

Michal
Comment 8 Michal Novotny 2010-08-17 09:07:02 EDT
Also, the information about whether you were using PV drivers or not is missing. I didn't use them at all.

Could you please try on a bare-metal system as I wrote in comment #7 ?

Thanks,
Michal
Comment 9 Michal Novotny 2010-08-17 10:30:58 EDT
This may be relevant as well: http://xenbits.xensource.com/staging/xen-unstable.hg?rev/a6f24be72f66

Michal
Comment 10 Michal Novotny 2010-08-17 10:39:40 EDT
Studying it a little more I saw an upstream patch that's necessary for CPU hotplug at [1] so I guess this is the one to be backported. I'll try although it's a hypervisor related thing.

Michal

[1] http://xenbits.xensource.com/staging/xen-unstable.hg?rev/625aa1547cb6
Comment 11 Michal Novotny 2010-08-17 11:39:24 EDT
(In reply to comment #10)
> Studying it a little more I saw an upstream patch that's necessary for CPU
> hotplug at [1] so I guess this is the one to be backported. I'll try although
> it's a hypervisor related thing.
> 
> Michal
> 
> [1] http://xenbits.xensource.com/staging/xen-unstable.hg?rev/625aa1547cb6

Well, there were some pieces of this patch missing in the latest hypervisor coding but unfortunately it was not working when applied. Based on the fact that the configuration had to be changed (new VCPU added) before guest reboot I did one more testing on the upstream version but it appears to be working as well so I think this may be hypervisor related after all.

Michal
Comment 12 shaochen 2010-08-17 21:39:47 EDT
(In reply to comment #8)
> Also, the information about whether you were using PV drivers or not is
> missing. I didn't use them at all.
> 
> Could you please try on a bare-metal system as I wrote in comment #7 ?
> 
> Thanks,
> Michal

I will test on a bara-metal system as you wrote.
Comment 13 shaochen 2010-08-18 03:30:21 EDT
(In reply to comment #8)
> Also, the information about whether you were using PV drivers or not is
> missing. I didn't use them at all.
> 
> Could you please try on a bare-metal system as I wrote in comment #7 ?
> 
> Thanks,
> Michal

Hi Michal, 
I am very sorry, because we haven't such test environment in laboratory. So I can't do the test on a bara-metal system as you wrote in comment #7.


About whether using XenPV drivers, please see below information for more details.

1) Without Xenpv drivers in Windows 2008 R2 x64 guest: Windows guests with multiple vcpus can restart smoothly.

2) With Xenpv drivers in Windows 2008 R2 x64 guest: Windows guests with multiple vcpus can't restart.

Add new xend.log to attachment.
Comment 14 shaochen 2010-08-18 03:31:06 EDT
Created attachment 439311 [details]
add new xend.log
Comment 15 Paolo Bonzini 2010-08-18 05:07:09 EDT
I also noticed this problem when doing my own WHQL test runs.

Can you test whether the bug also happens with AMD hardware?
Comment 16 Paolo Bonzini 2010-08-18 05:10:33 EDT
Michal, we already have the patch at c/s 13812 (comment #10), even though the code is slightly different.
Comment 17 Paolo Bonzini 2010-08-18 05:11:59 EDT
(In reply to comment #15)
> I also noticed this problem when doing my own WHQL test runs.
> 
> Can you test whether the bug also happens with AMD hardware?

Michal tested it and it's only on Intel hardware.
Comment 18 Paolo Bonzini 2010-08-18 05:18:53 EDT
This is the that patch should fix it, but it's very very large:

http://xenbits.xensource.com/xen-unstable.hg/rev/15388
Comment 19 Paolo Bonzini 2010-08-18 11:02:48 EDT
*** Bug 574123 has been marked as a duplicate of this bug. ***
Comment 21 Andrew Jones 2010-09-22 08:08:45 EDT
*** Bug 636421 has been marked as a duplicate of this bug. ***
Comment 24 Andrew Jones 2010-11-26 03:30:19 EST
*** Bug 657281 has been marked as a duplicate of this bug. ***
Comment 25 Kirby Zhou 2010-11-28 07:36:57 EST
RHEL-5.5 also has this bug, with:

kernel-xen-2.6.18-194.8.1.el5 and xen-3.0.3-105.el5_5.4
kernel-xen-2.6.18-194.17.1.el5 and xen-3.0.3-105.el5_5.5

but kernel-xen-2.6.18-194.3.1.el5 and xen-3.0.3-105.el5_5.3 seems no problem.
Comment 26 Paolo Bonzini 2010-11-29 05:11:45 EST
The bug may be harder to trigger in some conditions, but it has been analyzed quite thoroughly and it definitely _is_ present in all 5.5.z kernels.  Also, the userspace version does not matter as the bug is purely in the kernel.

The easiest way to trigger is to install a 32-bit Windows OS and switch VCPUs repeatedly from 1 to 2 and back.
Comment 29 RHEL Product and Program Management 2010-12-08 14:39:46 EST
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 33 errata-xmlrpc 2011-01-13 16:09:57 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html

Note You need to log in before you can comment on or make changes to this bug.