Hide Forgot
Description of problem: Occasionally show BSOD. QE can not 100% reproduce it. I pasted part analysis result of dump file. As this bug from acceptance test. So QE will attach full dump file when BOSD show again. Version-Release number of selected component (if applicable): 3.10.0-290.el7.x86_64 qemu-kvm-rhev-2.3.0-7.el7.x86_64 How reproducible: Occasionally Steps to Reproduce: 1. qemu command line /usr/libexec/qemu-kvm \ -name 'virt-tests-vm1' \ -sandbox off \ -machine q35 \ -nodefaults \ -vga qxl \ -device intel-hda,bus=pcie.0,addr=02 \ -device hda-duplex \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150704-133854-uQYe9gj9,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20150704-133854-uQYe9gj9,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150704-133854-uQYe9gj9,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -device virtio-serial-pci,id=virtio_serial_pci0,bus=pcie.0,addr=03 \ -chardev socket,id=devvs,path=/tmp/virtio_port-vs-20150704-133854-uQYe9gj9,server,nowait \ -device virtserialport,chardev=devvs,name=vs,id=vs,bus=virtio_serial_pci0.0 \ -chardev socket,id=seabioslog_id_20150704-133854-uQYe9gj9,path=/tmp/seabios-20150704-133854-uQYe9gj9,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20150704-133854-uQYe9gj9,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pcie.0,addr=04 \ -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/home/win2008-r2-64-virtio.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pcie.0,addr=05 \ -device virtio-net-pci,mac=9a:a4:a5:a6:a7:a8,id=idVwpA5h,vectors=4,netdev=idAMDjJz,bus=pcie.0,addr=06 \ -netdev tap,id=idAMDjJz,vhost=on \ -m 4096 \ -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \ -cpu 'Opteron_G3',hv_relaxed,+kvm_pv_unhalt,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -drive id=drive_cd1,if=none,snapshot=off,cache=none,aio=native,media=cdrom,file=/home/auto/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \ -device ide-cd,id=cd1,drive=drive_cd1,bootindex=1,bus=ide.0,unit=0 \ -drive id=drive_cd2,if=none,snapshot=off,cache=none,aio=native,media=cdrom,file=/home/auto/autotest-devel/client/tests/virt/shared/data/images/orig.iso \ -device ide-cd,id=cd2,drive=drive_cd2,bootindex=2,bus=ide.1,unit=0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -vnc :1 \ -monitor stdio 2. 3. Actual results: windbg analysis result ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* Use !analyze -v to get detailed debugging information. BugCheck 1E, {0, 0, 0, 0} Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE ) Followup: MachineOwner --------- 16.12: kd:x86> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* KMODE_EXCEPTION_NOT_HANDLED (1e) This is a very common bugcheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Arguments: Arg1: 0000000000000000, The exception code that was not handled Arg2: 0000000000000000, The address that the exception occurred at Arg3: 0000000000000000, Parameter 0 of the exception Arg4: 0000000000000000, Parameter 1 of the exception Debugging Details: ------------------ DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT BUGCHECK_STR: 0x1E CURRENT_IRQL: 0 ANALYSIS_VERSION: 6.3.9600.16520 (debuggers(dbg).140127-0329) amd64fre LAST_CONTROL_TRANSFER: from 0000000000000000 to 0000000000000000 STACK_TEXT: 00000000 00000000 00000000 00000000 00000000 0x0 STACK_COMMAND: kb SYMBOL_NAME: ANALYSIS_INCONCLUSIVE FOLLOWUP_NAME: MachineOwner MODULE_NAME: Unknown_Module IMAGE_NAME: Unknown_Image DEBUG_FLR_IMAGE_TIMESTAMP: 0 IMAGE_VERSION: BUCKET_ID: INVALID_KERNEL_CONTEXT FAILURE_BUCKET_ID: INVALID_KERNEL_CONTEXT ANALYSIS_SOURCE: KM FAILURE_ID_HASH_STRING: km:invalid_kernel_context FAILURE_ID_HASH: {ef5f68ed-c19c-e34b-48ec-8a37cd6f3937} Followup: MachineOwner --------- Expected results: Additional info:
Amnon, I am trying to reproduce it by auto testing. Will update result to bz once I reproduce it.
Created attachment 1101681 [details] Screenshot of the bsod.jpg I met the bug during our autotest acceptance test. The detailed sw versions are as follows: Host kernel: 3.10.0-327.3.1.el7.x86_64 Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-31.el7_2.4.x86_64 Virtio-Win: virtio-win-1.8.0-4.iso spice-server-0.12.4-15.el7.x86_64
Created attachment 1103034 [details] New screenshot for BSOD-12072015.png (In reply to Amnon Ilan from comment #11) > Is there a dump file? > Can you reproduce it with dump file? Following is the dump file and guest image that reproduced the bsod: win7 dump file: http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/rhel7/bug1241986/12072015-Memory.dmp win7 guest image: http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/rhel7/bug1241986/12072015-win7-64-sp1-virtio.qcow2 I could reproduce the attached BSOD with following the qemu cmd line on the same guest image win7-64-sp1-virtio.qcow2: /usr/libexec/qemu-kvm \ -name 'virt-tests-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga qxl \ -device intel-hda,bus=pci.0,addr=03 \ -device hda-duplex \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20151203-055253-G9hWBy9V,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20151203-055253-G9hWBy9V,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idzoJXK8 \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20151203-055253-G9hWBy9V,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20151203-055253-G9hWBy9V,path=/tmp/seabios-20151203-055253-G9hWBy9V,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20151203-055253-G9hWBy9V,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=04 \ -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/home/autotest/client/tests/virt/shared/data/images/win7-64-sp1-virtio.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=05,disable-legacy=off,disable-modern=on \ -device virtio-net-pci,mac=9a:c3:c4:c5:c6:c7,id=idUvUYlL,vectors=4,netdev=idEEyj6H,bus=pci.0,addr=06,disable-legacy=off,disable-modern=on \ -netdev tap,id=idEEyj6H,vhost=on \ -m 8192 \ -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \ -cpu 'Opteron_G3',hv_relaxed,+kvm_pv_unhalt,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/autotest/client/tests/virt/shared/data/isos/windows/winutils.iso \ -device ide-cd,id=cd1,drive=drive_cd1,bootindex=1,bus=ide.0,unit=0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm
(In reply to Gu Nini from comment #12) > Created attachment 1103034 [details] > New screenshot for BSOD-12072015.png > > (In reply to Amnon Ilan from comment #11) > > Is there a dump file? > > Can you reproduce it with dump file? > > Following is the dump file and guest image that reproduced the bsod: > > win7 dump file: > http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/rhel7/ > bug1241986/12072015-Memory.dmp > for this particular case the crash was triggered by netkvm driver: 0: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* IRQL_NOT_LESS_OR_EQUAL (a) An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If a kernel debugger is available get the stack backtrace. Arguments: Arg1: 0000000017400111, memory referenced Arg2: 0000000000000002, IRQL Arg3: 0000000000000000, bitfield : bit 0 : value 0 = read operation, 1 = write operation bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status) Arg4: fffff800026a27d5, address which referenced memory Debugging Details: ------------------ READ_ADDRESS: 0000000017400111 CURRENT_IRQL: 2 FAULTING_IP: nt!KeSetEvent+1e3 fffff800`026a27d5 488b00 mov rax,qword ptr [rax] DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT BUGCHECK_STR: 0xA PROCESS_NAME: System ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre DPC_STACK_BASE: FFFFF80003D53FB0 TRAP_FRAME: fffff80003d4d8a0 -- (.trap 0xfffff80003d4d8a0) NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=0000000017400111 rbx=0000000000000000 rcx=fffff880039b29d8 rdx=fffffa80081771a0 rsi=0000000000000000 rdi=0000000000000000 rip=fffff800026a27d5 rsp=fffff80003d4da30 rbp=0000000000000002 r8=0000000000000100 r9=0000000000000000 r10=0000000017400111 r11=0000000000000002 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz ac po cy nt!KeSetEvent+0x1e3: fffff800`026a27d5 488b00 mov rax,qword ptr [rax] ds:00000000`17400111=???????????????? Resetting default scope LAST_CONTROL_TRANSFER: from fffff8000269dbe9 to fffff8000269e640 STACK_TEXT: fffff800`03d4d758 fffff800`0269dbe9 : 00000000`0000000a 00000000`17400111 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx fffff800`03d4d760 fffff800`0269c860 : 00000000`00000000 fffff800`03d4d818 fffff8a0`03403000 fffff880`039b29d0 : nt!KiBugCheckDispatch+0x69 fffff800`03d4d8a0 fffff800`026a27d5 : fffff880`039b29d0 00000000`00000002 00000000`00000000 fffff800`02810e80 : nt!KiPageFault+0x260 fffff800`03d4da30 fffff880`0173b407 : fffff880`00000000 fffffa80`00000000 fffffa80`08177100 fffffa80`0846e200 : nt!KeSetEvent+0x1e3 fffff800`03d4daa0 fffff880`0436b870 : fffffa80`0846e200 00000000`00000000 00000000`00000000 fffffa80`0844e4f0 : ndis!NdisMPauseComplete+0x67 fffff800`03d4dad0 fffff880`043686d9 : fffffa80`0844e5e8 fffffa80`0844e530 fffffa80`0844e4f0 fffff880`04364acb : netkvm!OnSendPauseComplete+0x50 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\wlh\parandis6-driver.cpp @ 439] fffff800`03d4db00 fffff880`043621ed : e23be23a`00000000 00000000`00000001 00000000`00000000 00000000`00000000 : netkvm!CParaNdisTX::DoPendingTasks+0xcd [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\common\parandis-tx.cpp @ 622] fffff800`03d4dbb0 fffff880`0436d750 : fffffa80`0846e200 00000000`00000000 00000000`00000000 00000000`00000000 : netkvm!ParaNdis_DPCWorkBody+0xfd [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\common\parandis-common.cpp @ 1754] fffff800`03d4dbe0 fffff880`016adb51 : 0000000e`00000000 fffff800`03d4dd78 fffff800`02810e80 00000000`00000002 : netkvm!MiniportMSIInterruptDpc+0x2c [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\wlh\parandis6-impl.cpp @ 399] fffff800`03d4dc40 fffff800`026a9b1c : fffffa80`08435318 fffff800`00000000 00000000`00000000 fffff800`02810e80 : ndis!ndisInterruptDpc+0x151 fffff800`03d4dcd0 fffff800`0269636a : fffff800`02810e80 fffff800`0281ecc0 00000000`00000000 fffff880`016ada00 : nt!KiRetireDpcList+0x1bc fffff800`03d4dd80 00000000`00000000 : fffff800`03d4e000 fffff800`03d48000 fffff800`03d4dd40 00000000`00000000 : nt!KiIdleLoop+0x5a STACK_COMMAND: kb FOLLOWUP_IP: netkvm!OnSendPauseComplete+50 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\wlh\parandis6-driver.cpp @ 439] fffff880`0436b870 4883c420 add rsp,20h FAULTING_SOURCE_LINE: c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\wlh\parandis6-driver.cpp FAULTING_SOURCE_FILE: c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\netkvm\wlh\parandis6-driver.cpp FAULTING_SOURCE_LINE_NUMBER: 439 FAULTING_SOURCE_CODE: 435: // pause exit 436: ParaNdis_DebugHistory(pContext, hopSysPause, NULL, 0, 0, 0); 437: NdisMPauseComplete(pContext->MiniportHandle); 438: } > 439: } 440: 441: 442: /********************************************************** 443: Required NDIS handler 444: called at IRQL = PASSIVE_LEVEL SYMBOL_STACK_INDEX: 5 SYMBOL_NAME: netkvm!OnSendPauseComplete+50 FOLLOWUP_NAME: MachineOwner MODULE_NAME: netkvm IMAGE_NAME: netkvm.sys DEBUG_FLR_IMAGE_TIMESTAMP: 5600f3af FAILURE_BUCKET_ID: X64_0xA_netkvm!OnSendPauseComplete+50 BUCKET_ID: X64_0xA_netkvm!OnSendPauseComplete+50 ANALYSIS_SOURCE: KM FAILURE_ID_HASH_STRING: km:x64_0xa_netkvm!onsendpausecomplete+50 FAILURE_ID_HASH: {344b9418-3f44-d3cd-7f7a-bcf2e3647d8e} Followup: MachineOwner --------- 0: kd> dt netkvm!PARANDIS_ADAPTER 0xfffffa80`0846e200 +0x000 DriverHandle : 0xfffffa80`083de020 Void +0x008 MiniportHandle : 0xfffffa80`081771a0 Void +0x010 InterruptHandle : 0xfffffa80`08435000 Void +0x018 BufferListsPool : 0xfffffa80`08456480 Void +0x020 ResetEvent : _NDIS_EVENT +0x038 AdapterResources : _tagAdapterResources +0x058 pIoPortOffset : 0x00000000`0000c060 Void +0x060 IODevice : 0xfffffa80`0846e0a0 TypeVirtIODevice +0x068 LastTxCompletionTimeStamp : _LARGE_INTEGER 0x0 +0x070 LastInterruptTimeStamp : _LARGE_INTEGER 0x01d12d51`35d9f072 +0x078 u32HostFeatures : 0x79bfffe7 +0x07c u32GuestFeatures : 0x38139823 +0x080 bConnected : 0x1 '' +0x084 fCurrentLinkState : 1 ( MediaConnectStateConnected ) +0x088 bEnableInterruptHandlingDPC : 0x1 '' +0x089 bEnableInterruptChecking : 0 '' +0x08a bDoSupportPriority : 0x1 '' +0x08b bLinkDetectSupported : 0x1 '' +0x08c bGuestChecksumSupported : 0x1 '' +0x08d bControlQueueSupported : 0x1 '' +0x08e bUseMergedBuffers : 0x1 '' +0x08f bDoPublishIndices : 0x1 '' +0x090 bSurprizeRemoved : 0 '' +0x091 bUsingMSIX : 0x1 '' +0x092 bUseIndirect : 0x1 '' +0x093 bAnyLaypout : 0x1 '' +0x094 bHasHardwareFilters : 0x1 '' +0x095 bNoPauseOnSuspend : 0 '' +0x096 bFastSuspendInProcess : 0 '' +0x097 bResetInProgress : 0 '' +0x098 bCtrlMACAddrSupported : 0 '' +0x099 bCfgMACAddrSupported : 0x1 '' +0x09a bMultiQueue : 0 '' +0x09c nHardwareQueues : 1 +0x0a0 ulCurrentVlansFilterSet : 0 +0x0a4 MulticastData : _tagMulticastData +0x168 uNumberOfHandledRXPacketsInDPC : 0x3e8 +0x16c powerState : 1 ( NdisDeviceStateD0 ) +0x170 nPendingDPCs : 0n0 +0x174 counterDPCInside : 0n1 +0x178 bDPCInactive : 0n0 +0x17c ulPriorityVlanSetting : 3 +0x180 VlanId : 0 +0x188 ulFormalLinkSpeed : 0x00000002`540be400 +0x190 ulEnableWakeup : 0 +0x194 MaxPacketSize : _tagMaxPacketSize +0x1a8 ulUniqueID : 1 +0x1ac PermanentMacAddress : [6] "???" +0x1b2 CurrentMacAddress : [6] "???" +0x1b8 PacketFilter : 0 +0x1bc DummyLookAhead : 0 +0x1c0 nDetectedStoppedTx : 0 +0x1c4 nDetectedInactivity : 0 +0x1c8 nVirtioHeaderSize : 0xc +0x1d0 Statistics : _NDIS_STATISTICS_INFO +0x268 extraStatistics : _tagPARANDIS_ADAPTER::<unnamed-type-extraStatistics> +0x288 Counters : _tagOurCounters +0x2a0 Limits : _tagOurCounters +0x2b8 SendState : 0 ( srsDisabled ) +0x2bc ReceiveState : 0 ( srsDisabled ) +0x2c0 SendPauseCompletionProc : (null) +0x2c8 ReceivePauseCompletionProc : (null) +0x2d0 m_PauseLock : CNdisRWLock +0x2d8 m_CompletionLock : _NDIS_SPIN_LOCK +0x2e8 m_CompletionLockCreated : 1 +0x2ec m_rxPacketsOutsideRing : CNdisRefCounter +0x2f0 maxFreeTxDescriptors : 0x400 +0x2f4 NetMaxReceiveBuffers : 0x100 +0x2f8 nPnpEventIndex : 1 +0x2fc PnpEvents : [16] 5 ( NdisDevicePnPEventPowerProfileChanged ) +0x33c Offload : _tagOffloadSettings +0x34c InitialOffloadParameters : _NDIS_OFFLOAD_PARAMETERS +0x368 ReceiveQueues : [16] _tagPARANDIS_RECEIVE_QUEUE +0x5e8 ReceiveQueuesInitialized : 0x1 '' +0x5f0 CXPath : CParaNdisCX +0x6b0 bCXPathAllocated : 0x1 '' +0x6b1 bCXPathCreated : 0x1 '' +0x6b8 pPathBundles : 0xfffffa80`0844e4f0 CPUPathesBundle +0x6c0 nPathBundles : 1 +0x6c8 RSS2QueueMap : (null) +0x6d0 RSS2QueueLength : 0 +0x6d8 pMSIXInfoTable : 0xfffffa80`08436e20 _IO_INTERRUPT_MESSAGE_INFO +0x6e0 DmaHandle : 0xfffffa80`08436d70 Void +0x6e8 ulIrqReceived : 1 +0x6ec ReportedOffloadCapabilities : _NDIS_OFFLOAD +0x77c ReportedOffloadConfiguration : _NDIS_OFFLOAD +0x80c bOffloadv4Enabled : 0 '' +0x80d bOffloadv6Enabled : 0 '' +0x80e bDeviceInitialized : 0x1 '' +0x80f bRSSOffloadSupported : 0x1 '' +0x810 bRSSInitialized : 0x1 '' +0x814 RSSCapabilities : _NDIS_RECEIVE_SCALE_CAPABILITIES +0x828 RSSParameters : _tagPARANDIS_RSS_PARAMS +0xdf8 RSSMaxQueuesNumber : 8 '' I'm going to ask Yan to look into the problem.
Yan, Could you please take a look at comment13? see if can find root reason. Thanks.
(In reply to FuXiangChun from comment #14) > Yan, > Could you please take a look at comment13? see if can find root reason. > Thanks. I am working on fixing this bug. This is a race condition on NIC pause.
*** Bug 1274125 has been marked as a duplicate of this bug. ***
Reproduce this bug on virtio-win-1.8.0-4 Verified this bug on virtio-win-prewhql-118 kernel-3.10.0-327.el7.x86_64 qemu-kvm-tools-rhev-2.3.0-1.el7.x86_64 Steps as comment#0 reboot 100 times , not hit BSOD Above all, this bug has been fixed. Thanks Yu Wang
According comment#18, change status to "verified" Thanks Yu Wang
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2609.html