669316 – [WHQL] BSOD occurs when running "CHAOS-Concurrent Hardware And OS test"

Bug 669316 - [WHQL] BSOD occurs when running "CHAOS-Concurrent Hardware And OS test"

Summary: [WHQL] BSOD occurs when running "CHAOS-Concurrent Hardware And OS test"

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	virtio-win
Sub Component:
Version:	6.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Vadim Rozenfeld
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2011-01-13 10:25 UTC by Qunfang Zhang
Modified:	2013-01-09 23:29 UTC (History)
CC List:	6 users (show)
Fixed In Version:	virtio-win-1.1.19
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2011-05-19 14:31:42 UTC
Target Upstream Version:

Attachments	(Terms of Use)
BSOD when reboot (22.74 KB, image/png) 2011-01-13 10:25 UTC, Qunfang Zhang	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2011:0782	0	normal	SHIPPED_LIVE	virtio-win bug fix and enhancement update	2011-05-18 18:08:21 UTC

Description Qunfang Zhang 2011-01-13 10:25:53 UTC

Created attachment 473286 [details]
BSOD when reboot

Description of problem:
When running the child job "disable driver verifier" in the job CHAOS, a BSOD always happens during reboot guest.
Can not provide the memory dump file due to Bug 669314. Screenshot will be attached.

Version-Release number of selected component (if applicable):
virtio-win-1.1.17
kernel-2.6.32-95.el6.x86_64
qemu-kvm-0.12.1.2-2.129.el6.x86_64
seabios-0.5.1-3.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Run job "CHAOS-Concurrent Hardware And OS test", for win7-32, maybe easily to reproduce.
2.
3.
  
Actual results:
BSOD happens during reboot when running the job.

Expected results:
No BSOD happens when running this job.

Additional info:

Comment 2 Dor Laor 2011-01-17 10:18:39 UTC

Is this a must requirement by WHQL?
Any logs (dmesg) on the host?

Comment 3 Qunfang Zhang 2011-01-17 10:28:16 UTC

(In reply to comment #2)
> Is this a must requirement by WHQL?
It will not block passing whql, BSOD happened during reboot guest. So I need to do "system_reset" on qemu monitor then the job will continue.

> Any logs (dmesg) on the host?
No additional logs.

Comment 4 Vadim Rozenfeld 2011-01-22 20:08:42 UTC

Please try the latest driver from:
http://download.lab.bos.redhat.com/devel/RHEV/virtio-win/1.1.18/
Vadim.

Comment 5 dawu 2011-01-25 09:35:30 UTC

Verified this issue with the latest driver 1.1.18, it still reproduce, following is the dump file analysis:

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_POWER_STATE_FAILURE (9f)
A driver is causing an inconsistent power state.
Arguments:
Arg1: 0000000000000004, The power transition timed out waiting to synchronize with the Pnp
	subsystem.
Arg2: 0000000000000258, Timeout in seconds.
Arg3: fffffa80018e2680
Arg4: fffff80000b9c510

Debugging Details:
------------------


Could not read faulting driver name

DRVPOWERSTATE_SUBCODE:  4

DRIVER_OBJECT: fffff80000b9c510

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x9F

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from fffff80002720376 to fffff8000268d740

STACK_TEXT:  
fffff800`00b9c4d8 fffff800`02720376 : 00000000`0000009f 00000000`00000004 00000000`00000258 fffffa80`018e2680 : nt!KeBugCheckEx
fffff800`00b9c4e0 fffff800`028cf40c : fffff800`00000000 fffffa80`00000000 00000000`00000200 fffff800`026921fa : nt!PnpBugcheckPowerTimeout+0x76
fffff800`00b9c540 fffff800`0269929e : 00000000`00000001 00000000`00000003 00000000`00000000 00000000`00000000 : nt!PopBuildDeviceNotifyListWatchdog+0x1c
fffff800`00b9c570 fffff800`02698dd6 : fffffa80`02685408 fffffa80`02685408 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
fffff800`00b9c5e0 fffff800`026994be : 00000003`6a93bc2e fffff800`00b9cc58 00000000`00016ed3 fffff800`0280bce8 : nt!KiProcessExpiredTimerList+0xc6
fffff800`00b9cc30 fffff800`02698cb7 : 00000001`391c20c6 00000001`00016ed3 00000001`391c2048 00000000`000000d3 : nt!KiTimerExpiration+0x1be
fffff800`00b9ccd0 fffff800`02695eea : fffff800`02807e80 fffff800`02815c40 00000000`00000000 fffff880`01026f44 : nt!KiRetireDpcList+0x277
fffff800`00b9cd80 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cd40 00000000`00000000 : nt!KiIdleLoop+0x5a


STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!PnpBugcheckPowerTimeout+76
fffff800`02720376 cc              int     3

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  nt!PnpBugcheckPowerTimeout+76

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  4c1c44a9

FAILURE_BUCKET_ID:  X64_0x9F_VRF_nt!PnpBugcheckPowerTimeout+76

BUCKET_ID:  X64_0x9F_VRF_nt!PnpBugcheckPowerTimeout+76

Followup: MachineOwner
---------

Please refer to http://10.66.65.33/dump-file/MEMORY-bug670713.DMP for details.

Comment 6 Vadim Rozenfeld 2011-01-25 13:45:08 UTC

Did it work with virtio-win-1.1.16?
Can you please post qemu command line as well?
Thank you,
Vadim.

Comment 7 Qunfang Zhang 2011-01-26 01:58:45 UTC

(In reply to comment #6)
> Did it work with virtio-win-1.1.16?
Vadim, Yes, it worked with virtio-win-1.1.16. We did not meet this BSOD during running this job.

> Can you please post qemu command line as well?
CLI:
/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu cpu64-rhel6,+x2apic,family=0xf -usbdevice tablet -drive file=win7-64-blk.raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:10:40:32:72:68,bus=pci.0,addr=0x4 -boot c -uuid 6afdacf7-71a1-4250-9a02-451803d9bcc9 -rtc-td-hack -no-kvm-pit-reinjection -monitor stdio -name win7-64-blk -vnc :10 -drive file=disk1.raw,if=none,id=drive-virtio1,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio1,id=virtio-blk-pci1 -drive file=disk2.raw,if=none,id=drive-virtio2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio2,id=virtio-blk-pci2 -drive file=disk3.raw,if=none,id=drive-virtio3,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio3,id=virtio-blk-pci3 -drive file=disk4.raw,if=none,id=drive-virtio4,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio4,id=virtio-blk-pci4

> Thank you,
> Vadim.

Comment 8 Vadim Rozenfeld 2011-02-12 18:16:06 UTC

******************************************************************************
> 
> DRIVER_POWER_STATE_FAILURE (9f)
> A driver is causing an inconsistent power state.
> Arguments:
> Arg1: 0000000000000004, The power transition timed out waiting to synchronize
> with the Pnp
>  subsystem.
> Arg2: 0000000000000258, Timeout in seconds.
> Arg3: fffffa80018e2680
> Arg4: fffff80000b9c510
> 
> Debugging Details:
> ------------------
> 
> 
> Could not read faulting driver name
> 
> DRVPOWERSTATE_SUBCODE:  4
> 
> DRIVER_OBJECT: fffff80000b9c510
> 
> DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
> 
> BUGCHECK_STR:  0x9F
> 
> PROCESS_NAME:  System
> 
> CURRENT_IRQL:  2
> 
> LAST_CONTROL_TRANSFER:  from fffff80002720376 to fffff8000268d740
> 
> STACK_TEXT:  
> fffff800`00b9c4d8 fffff800`02720376 : 00000000`0000009f 00000000`00000004
> 00000000`00000258 fffffa80`018e2680 : nt!KeBugCheckEx
> fffff800`00b9c4e0 fffff800`028cf40c : fffff800`00000000 fffffa80`00000000
> 00000000`00000200 fffff800`026921fa : nt!PnpBugcheckPowerTimeout+0x76
> fffff800`00b9c540 fffff800`0269929e : 00000000`00000001 00000000`00000003
> 00000000`00000000 00000000`00000000 : nt!PopBuildDeviceNotifyListWatchdog+0x1c
> fffff800`00b9c570 fffff800`02698dd6 : fffffa80`02685408 fffffa80`02685408
> 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
> fffff800`00b9c5e0 fffff800`026994be : 00000003`6a93bc2e fffff800`00b9cc58
> 00000000`00016ed3 fffff800`0280bce8 : nt!KiProcessExpiredTimerList+0xc6
> fffff800`00b9cc30 fffff800`02698cb7 : 00000001`391c20c6 00000001`00016ed3
> 00000001`391c2048 00000000`000000d3 : nt!KiTimerExpiration+0x1be
> fffff800`00b9ccd0 fffff800`02695eea : fffff800`02807e80 fffff800`02815c40
> 00000000`00000000 fffff880`01026f44 : nt!KiRetireDpcList+0x277
> fffff800`00b9cd80 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000
> fffff800`00b9cd40 00000000`00000000 : nt!KiIdleLoop+0x5a


> Please refer to http://10.66.65.33/dump-file/MEMORY-bug670713.DMP for details.

Unfortunately, this dump file doesn't match the analyze report above. 
Could you please give me the location of the right one?

One more question. Is it reproducible on all platforms or only on Win7?
In case, if it is a Win7 bug only, could you try virtio-win-1.1.18 W2K8 drivers instead of Win7 ones?

Best regards,
Vadim.

Comment 9 Qunfang Zhang 2011-02-15 06:53:18 UTC

(In reply to comment #8)
> ******************************************************************************
> > 
> > DRIVER_POWER_STATE_FAILURE (9f)
> > A driver is causing an inconsistent power state.
> > Arguments:
> > Arg1: 0000000000000004, The power transition timed out waiting to synchronize
> > with the Pnp
> >  subsystem.
> > Arg2: 0000000000000258, Timeout in seconds.
> > Arg3: fffffa80018e2680
> > Arg4: fffff80000b9c510
> > 
> > Debugging Details:
> > ------------------
> > 
> > 
> > Could not read faulting driver name
> > 
> > DRVPOWERSTATE_SUBCODE:  4
> > 
> > DRIVER_OBJECT: fffff80000b9c510
> > 
> > DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
> > 
> > BUGCHECK_STR:  0x9F
> > 
> > PROCESS_NAME:  System
> > 
> > CURRENT_IRQL:  2
> > 
> > LAST_CONTROL_TRANSFER:  from fffff80002720376 to fffff8000268d740
> > 
> > STACK_TEXT:  
> > fffff800`00b9c4d8 fffff800`02720376 : 00000000`0000009f 00000000`00000004
> > 00000000`00000258 fffffa80`018e2680 : nt!KeBugCheckEx
> > fffff800`00b9c4e0 fffff800`028cf40c : fffff800`00000000 fffffa80`00000000
> > 00000000`00000200 fffff800`026921fa : nt!PnpBugcheckPowerTimeout+0x76
> > fffff800`00b9c540 fffff800`0269929e : 00000000`00000001 00000000`00000003
> > 00000000`00000000 00000000`00000000 : nt!PopBuildDeviceNotifyListWatchdog+0x1c
> > fffff800`00b9c570 fffff800`02698dd6 : fffffa80`02685408 fffffa80`02685408
> > 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
> > fffff800`00b9c5e0 fffff800`026994be : 00000003`6a93bc2e fffff800`00b9cc58
> > 00000000`00016ed3 fffff800`0280bce8 : nt!KiProcessExpiredTimerList+0xc6
> > fffff800`00b9cc30 fffff800`02698cb7 : 00000001`391c20c6 00000001`00016ed3
> > 00000001`391c2048 00000000`000000d3 : nt!KiTimerExpiration+0x1be
> > fffff800`00b9ccd0 fffff800`02695eea : fffff800`02807e80 fffff800`02815c40
> > 00000000`00000000 fffff880`01026f44 : nt!KiRetireDpcList+0x277
> > fffff800`00b9cd80 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000
> > fffff800`00b9cd40 00000000`00000000 : nt!KiIdleLoop+0x5a
> 
> 
> > Please refer to http://10.66.65.33/dump-file/MEMORY-bug670713.DMP for details.
> 
> Unfortunately, this dump file doesn't match the analyze report above. 
> Could you please give me the location of the right one?
> 
> One more question. Is it reproducible on all platforms or only on Win7?
> In case, if it is a Win7 bug only, could you try virtio-win-1.1.18 W2K8 drivers
> instead of Win7 ones?
> 
> Best regards,
> Vadim.

Hi Vadim
This issue only happens on win7 before, and I re-test with W2K8 driver just now, it passed.

Comment 10 Vadim Rozenfeld 2011-02-15 12:29:22 UTC

(In reply to comment #9)
> (In reply to comment #8)
> > ******************************************************************************
> > > 
> > > DRIVER_POWER_STATE_FAILURE (9f)
> > > A driver is causing an inconsistent power state.
> > > Arguments:
> > > Arg1: 0000000000000004, The power transition timed out waiting to synchronize
> > > with the Pnp
> > >  subsystem.
> > > Arg2: 0000000000000258, Timeout in seconds.
> > > Arg3: fffffa80018e2680
> > > Arg4: fffff80000b9c510
> > > 
> > > Debugging Details:
> > > ------------------
> > > 
> > > 
> > > Could not read faulting driver name
> > > 
> > > DRVPOWERSTATE_SUBCODE:  4
> > > 
> > > DRIVER_OBJECT: fffff80000b9c510
> > > 
> > > DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
> > > 
> > > BUGCHECK_STR:  0x9F
> > > 
> > > PROCESS_NAME:  System
> > > 
> > > CURRENT_IRQL:  2
> > > 
> > > LAST_CONTROL_TRANSFER:  from fffff80002720376 to fffff8000268d740
> > > 
> > > STACK_TEXT:  
> > > fffff800`00b9c4d8 fffff800`02720376 : 00000000`0000009f 00000000`00000004
> > > 00000000`00000258 fffffa80`018e2680 : nt!KeBugCheckEx
> > > fffff800`00b9c4e0 fffff800`028cf40c : fffff800`00000000 fffffa80`00000000
> > > 00000000`00000200 fffff800`026921fa : nt!PnpBugcheckPowerTimeout+0x76
> > > fffff800`00b9c540 fffff800`0269929e : 00000000`00000001 00000000`00000003
> > > 00000000`00000000 00000000`00000000 : nt!PopBuildDeviceNotifyListWatchdog+0x1c
> > > fffff800`00b9c570 fffff800`02698dd6 : fffffa80`02685408 fffffa80`02685408
> > > 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
> > > fffff800`00b9c5e0 fffff800`026994be : 00000003`6a93bc2e fffff800`00b9cc58
> > > 00000000`00016ed3 fffff800`0280bce8 : nt!KiProcessExpiredTimerList+0xc6
> > > fffff800`00b9cc30 fffff800`02698cb7 : 00000001`391c20c6 00000001`00016ed3
> > > 00000001`391c2048 00000000`000000d3 : nt!KiTimerExpiration+0x1be
> > > fffff800`00b9ccd0 fffff800`02695eea : fffff800`02807e80 fffff800`02815c40
> > > 00000000`00000000 fffff880`01026f44 : nt!KiRetireDpcList+0x277
> > > fffff800`00b9cd80 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000
> > > fffff800`00b9cd40 00000000`00000000 : nt!KiIdleLoop+0x5a
> > 
> > 
> > > Please refer to http://10.66.65.33/dump-file/MEMORY-bug670713.DMP for details.
> > 
> > Unfortunately, this dump file doesn't match the analyze report above. 
> > Could you please give me the location of the right one?
> > 
> > One more question. Is it reproducible on all platforms or only on Win7?
> > In case, if it is a Win7 bug only, could you try virtio-win-1.1.18 W2K8 drivers
> > instead of Win7 ones?
> > 
> > Best regards,
> > Vadim.
> 
> Hi Vadim
> This issue only happens on win7 before, and I re-test with W2K8 driver just
> now, it passed.

Great,
Now the problem should be fixed also in Win7 drivers.
Could you please try the latest ones:
http://download.lab.bos.redhat.com/devel/RHEV/virtio-win/1.1.19/

Best regards,
Vadim

Comment 11 dawu 2011-02-22 11:11:18 UTC

Verify this issue with build virtio-win-1-1-19, it does not reproduce,

following is the environment details:
virtio-1-1-19
qemu-kvm-0.12.1.2-2.146.el6.x86_64
kernel-2.6.32-113.el6.x86_64

Comment 13 errata-xmlrpc 2011-05-19 14:31:42 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0782.html

Note You need to log in before you can comment on or make changes to this bug.