Bug 850740 - win2008 guest BSOD when host has stress
win2008 guest BSOD when host has stress
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm (Show other bugs)
5.9
Unspecified Unspecified
high Severity medium
: rc
: ---
Assigned To: Virtualization Maintenance
Virtualization Bugs
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-22 05:43 EDT by Shaolong Hu
Modified: 2012-10-17 22:15 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-23 07:15:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Shaolong Hu 2012-08-22 05:43:34 EDT
Description of problem:
------------------------
win2008 guest get BSOD when adding stress to host.


Version-Release number of selected component (if applicable):
--------------------------------------------------------------
kvm-83-259.el5
stress-1.0.2-1.el5.rf


How reproducible:
-----------------
100%


Steps to Reproduce:
---------------------
1. turn swap off:

#swapoff -a

2. boot a win2008 guest, in guest add stress with tool "Heavyload" (it's free, please google it), stress cpu, memory and i/o:

#/usr/libexec/qemu-kvm -no-hpet -rtc-td-hack -smp 4 -m 4G -name win2008 -uuid 31e71eea-d178-4988-89da-25b2e4484001 -drive file=win2008-64-virtio.qcow2,format=qcow2,cache=off,index=0,boot=on,media=disk,if=virtio -monitor stdio -usbdevice tablet -net nic,vlan=0,model=virtio -net tap,vlan=0,ifname=net1,script=/etc/qemu-ifup -M rhel5.6.0 -vnc :10

3. after guest reboot, there is about 2G memory left (host has 8G), add stress to host with:

#stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M

4. guest got BSOD, at this time:

[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7451       5160       2290          0        111        693
-/+ buffers/cache:       4355       3096
Swap:            0          0          0

5. BSOD code 101:

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************



CLOCK_WATCHDOG_TIMEOUT (101)

An expected clock interrupt was not received on a secondary processor in an

MP system within the allocated interval. This indicates that the specified

processor is hung and not processing interrupts.

Arguments:

Arg1: 0000000000000030, Clock interrupt time out interval in nominal clock ticks.

Arg2: 0000000000000000, 0.

Arg3: fffffa60005ec180, The PRCB address of the hung processor.

Arg4: 0000000000000001, 0.



Debugging Details:

------------------



Page 130577 not present in the dump file. Type ".hh dbgerr004" for details

Page ba809 not present in the dump file. Type ".hh dbgerr004" for details

PEB is paged out (Peb.Ldr = 00000000`7efdf018).  Type ".hh dbgerr001" for details

PEB is paged out (Peb.Ldr = 00000000`7efdf018).  Type ".hh dbgerr001" for details



BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_4_PROC



DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT



PROCESS_NAME:  HeavyLoad.exe



CURRENT_IRQL:  d



STACK_TEXT:  

fffffa60`04df7b78 fffff800`016603a0 : 00000000`00000101 00000000`00000030 00000000`00000000 fffffa60`005ec180 : nt!KeBugCheckEx

fffffa60`04df7b80 fffff800`0169dd4a : 00000000`00000000 fffffa60`04df7ca0 00000000`0400ff40 fffff800`01631320 : nt! ?? ::FNODOBFM::`string'+0x2de4

fffffa60`04df7bc0 fffff800`016148af : 00000000`028a12d8 fffffa60`04df7ca0 fffff800`01631320 00000000`0400ff40 : nt!KeUpdateSystemTime+0xea

fffffa60`04df7bf0 fffff800`0169d50d : 00000000`028a12d8 fffff800`01631320 00000000`00001f7b fffffa80`04047900 : hal!HalpRtcClockInterrupt+0x127

fffffa60`04df7c20 00000000`00409557 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiInterruptDispatchNoLock+0x14d

00000000`03fe8180 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x409557





STACK_COMMAND:  kb



SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE



FOLLOWUP_NAME:  MachineOwner



MODULE_NAME: Unknown_Module



IMAGE_NAME:  Unknown_Image



DEBUG_FLR_IMAGE_TIMESTAMP:  0



FAILURE_BUCKET_ID:  X64_CLOCK_WATCHDOG_TIMEOUT_4_PROC_ANALYSIS_INCONCLUSIVE



BUCKET_ID:  X64_CLOCK_WATCHDOG_TIMEOUT_4_PROC_ANALYSIS_INCONCLUSIVE



Followup: MachineOwner

---------
Comment 1 Shaolong Hu 2012-08-22 05:47:26 EDT
we have met this before:

Bug 717882 - BSOD occurs when start 4 128GB windows guest on 512 GB host

it happens when there is memory overcommit, but this one happens when there is enough memory.


I am testing timer device on rhel5.9 host with win2008 guest, add stress to host is a basic step, so this is a test blocker.
Comment 2 Shaolong Hu 2012-08-22 05:50:44 EDT
guest is win2008 64 sp2 with latest virtio-win-1.0.3-1.52454.el5
Comment 3 Shaolong Hu 2012-08-22 06:00:45 EDT
btw, BSOD happens even do not add stress in guest with Heavyload, same BSOD code.
Comment 5 Ronen Hod 2012-08-23 03:38:13 EDT
(In reply to comment #1)
> we have met this before:
> 
> Bug 717882 - BSOD occurs when start 4 128GB windows guest on 512 GB host
> 
> it happens when there is memory overcommit, but this one happens when there
> is enough memory.
> 
> 
> I am testing timer device on rhel5.9 host with win2008 guest, add stress to
> host is a basic step, so this is a test blocker.

We know that RHEL5 host scheduler is inferior to RHEL6 scheduler, and this is not going to change. As a result processes (VCPUs) are not scheduled to run, and the guest BSOD.
I believe that you will run into the same issue with previous RHEL versions. It will probably happen with RHEL6 at some stress level.
Would it be possible to lower the stress level for the time-keeping test?

Thanks, Ronen.
Comment 6 Shaolong Hu 2012-08-23 06:35:28 EDT
(In reply to comment #5)

> We know that RHEL5 host scheduler is inferior to RHEL6 scheduler, and this
> is not going to change. As a result processes (VCPUs) are not scheduled to
> run, and the guest BSOD.
> I believe that you will run into the same issue with previous RHEL versions.
> It will probably happen with RHEL6 at some stress level.
> Would it be possible to lower the stress level for the time-keeping test?
> 
> Thanks, Ronen.

I did a little more test:

when use the tool "stress":

--cpu only add user space stress
--io and --vm only add kernel space stress

single user space or kernel space stress won't cause BSOD no matter how large the parameter is, only when there is high stress on both kernel and user space, causing frequent user/kernel space switching, BSOD happens:

single "--vm 1 --vm-bytes 128M" will give 99% system cpu usage on a 4 cores host, when add --cpu X, BSOD happens until the X up to 4.

I think in our RHEV-M product, we may want to warn users about this situation, some advice will be like save one or certain portion of physical cpu in host, or do not bind qemu-kvm process and vcpu to same pcpu, if i understand it correctly.

I will use some thing like "--cpu 3 --vm 4 --vm-bytes 128M" to do my test, i think it works for the run, thanks for specification and feel free to close this one.


Shaolong
Comment 7 Ronen Hod 2012-08-23 07:15:22 EDT
Shaolong,

Thanks for the additional tests and interesting results.
I am hesitant about adding specific instructions about how to avoid stress situations, since there are too many possible setups of VMs running on a single host doing all kind of things.
Since the issue is not new, and it is no more a test-blocker, I am closing this bug.

Ronen.
Comment 8 Shaolong Hu 2012-10-17 22:15:17 EDT
Just a FYI, test suggest this situation exists in latest 5.8.z, so this is not regression.

Test with:

2.6.18-308.14.1.el5
kvm-83-249.el5_8.5

Note You need to log in before you can comment on or make changes to this bug.