Bug 882795

Summary: [virtio-win][serial]BSOD occurs during transferring data from guest to host during s4
Product: Red Hat Enterprise Linux 6 Reporter: Mike Cao <bcao>
Component: virtio-winAssignee: Gal Hammer <ghammer>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: acathrow, bcao, bsarathy, dfleytma, juzhang, lijin, mdeng, michen, rhod, vrozenfe
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virtio-win-prewhql-0.1-51 Doc Type: Bug Fix
Doc Text:
Cause: Switching system Hibernate Mode(S4) while transferring data over virtio-serial device. Consequence: System fails into BSOD Fix: in virtio-serial driver. Changes in lock acquire/release logic. Result: System can be switched to S4 state without BSOD, after applying this fix.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-22 00:00:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Fix for the bug based on private repository from git.engineering none

Description Mike Cao 2012-12-03 06:34:53 UTC 
Description of problem:



Version-Release number of selected component (if applicable):
2.6.32-345.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.337.el6.x86_64
virtio-win-prewhql-49
win7-32bit guest

How reproducible:
2/2

Steps to Reproduce:
1.Start Guest with virtio-serial-pci
CLI:/usr/libexec/qemu-kvm -M rhel6.4.0 -m 2G -smp 2 -cpu host -usbdevice tablet -drive file=win7-32.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,werror=stop,rerror=stop,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:20:51:12:28:10,bus=pci.0,addr=0x4,id=net0 -boot c -uuid 813ef818-efd6-4031-9540-59f8c01d66e6 -rtc-td-hack -no-kvm-pit-reinjection -name win7-32 -vnc :4 -vga std -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=4,bus=pci.0 -chardev socket,path=/tmp/tt,server,nowait,id=channel0 -device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0 -monitor stdio -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=0 -cdrom /home/en_windows_7_ultimate_with_sp1_x86_dvd_619077.iso
2.on the host ,open the receiving side 
#for((;;)); do python serial-host-receive.py /tmp/tt; done
3.in the guest ,tranferring data via virtserialport
#for ((;;)); do python serial-host-receive.py com.redhat.rhevm.vdsm;done
4.doing hibernate in the guest
  
Actual results:
guest BSOD

Expected results:
No BSOD occurs


Additional info:
Comment 3 Mike Cao 2012-12-03 07:39:40 UTC 
S3 hit the same issue during transferring data from guest to host as well
Comment 4 Vadim Rozenfeld 2012-12-03 08:13:16 UTC 
Hi Gal,
Could you please take a look at this.
It seems to be regression introduced by e8a1de0760cc95332da52e681303031629015cdb
because it reverted some stuff fixed in 
c93ef0adfb6bd68e6cc730ebe6edf34972ee6bd3 :
Bug 806223 - [virtio-win][serial] guest BSOD when doing s3 while virtio-serial in use. Bug 714908 - [virtio-win][serial] virtio-serial is disfunctional after S3 Bug 814896 - [virtio-win][serial] virtio-serial does not work after s4 in win7-32/64 bit guests Bug 823818 - [virtio-win][compatibilty]Guest freeze After terminating the virtserialport receiving side in the host

Thank you,
Vadim.
Comment 5 Mike Cao 2012-12-03 08:18:59 UTC 
seems this is a known issue referring to Bug 806223
Comment 6 Mike Cao 2012-12-03 08:28:12 UTC 
also hit this issue when tranferring data from host to guest during s3
Comment 7 Vadim Rozenfeld 2012-12-03 09:15:54 UTC 
(In reply to comment #5)
> seems this is a known issue referring to Bug 806223

Hi Mike,
what about 806223 and 714908? Hope they are still fine.
Best regards,
Vadim.
Comment 8 Mike Cao 2012-12-03 09:26:29 UTC 
(In reply to comment #7)
> (In reply to comment #5)
> > seems this is a known issue referring to Bug 806223
> 
> Hi Mike,
> what about 806223 and 714908? Hope they are still fine.
> Best regards,
> Vadim.

I think this bug is dup of Bug 806223 ,and the status of Bug 806223 is post and product flag is rhel6.5.0 ,I never know it has been fixed ever.
Re-tried https://bugzilla.redhat.com/show_bug.cgi?id=714908 ,This issue works to me ,no regression found 

Mike
Comment 9 Gal Hammer 2012-12-05 15:10:41 UTC 
(In reply to comment #4)
> Hi Gal,
> Could you please take a look at this.
> It seems to be regression introduced by
> e8a1de0760cc95332da52e681303031629015cdb
> because it reverted some stuff fixed in 
> c93ef0adfb6bd68e6cc730ebe6edf34972ee6bd3 :
> Bug 806223 - [virtio-win][serial] guest BSOD when doing s3 while
> virtio-serial in use. Bug 714908 - [virtio-win][serial] virtio-serial is
> disfunctional after S3 Bug 814896 - [virtio-win][serial] virtio-serial does
> not work after s4 in win7-32/64 bit guests Bug 823818 -
> [virtio-win][compatibilty]Guest freeze After terminating the virtserialport
> receiving side in the host
> 
> Thank you,
> Vadim.

The bug occur on function VIOSerialPortCreate. The user open (create) the device after VIOSerialEvtDeviceD0Exit was called and return. The VIOSerialPortCreate function tries to notify the host that the port is open using VIOSerialSendCtrlMsg. But since the device is powered down at this stage pContext->c_ovq is NULL and I believe that the function call vq->vq_ops->add_buf crash the system.

The proper solution would be to tell WDF to disable the file handle to device. However, I don't know how to do it yet.

The quick solution would be to see if the device is active and if not fail the create request.

Vadim? What do you think?
Comment 10 Vadim Rozenfeld 2012-12-06 09:54:21 UTC 
(In reply to comment #9)
> (In reply to comment #4)
> > Hi Gal,
> > Could you please take a look at this.
> > It seems to be regression introduced by
> > e8a1de0760cc95332da52e681303031629015cdb
> > because it reverted some stuff fixed in 
> > c93ef0adfb6bd68e6cc730ebe6edf34972ee6bd3 :
> > Bug 806223 - [virtio-win][serial] guest BSOD when doing s3 while
> > virtio-serial in use. Bug 714908 - [virtio-win][serial] virtio-serial is
> > disfunctional after S3 Bug 814896 - [virtio-win][serial] virtio-serial does
> > not work after s4 in win7-32/64 bit guests Bug 823818 -
> > [virtio-win][compatibilty]Guest freeze After terminating the virtserialport
> > receiving side in the host
> > 
> > Thank you,
> > Vadim.
> 
> The bug occur on function VIOSerialPortCreate. The user open (create) the
> device after VIOSerialEvtDeviceD0Exit was called and return. The
That's strange. Do you mean that even though device is leaving D0 state and
the system is in transition from S0 to S4 we still continue receiving CreateFile requests from user mode application? If it's true, we must be able to hit the same problem during surprise removal.
 
> VIOSerialPortCreate function tries to notify the host that the port is open
> using VIOSerialSendCtrlMsg. But since the device is powered down at this
> stage pContext->c_ovq is NULL and I believe that the function call
> vq->vq_ops->add_buf crash the system.
> 
> The proper solution would be to tell WDF to disable the file handle to
> device. However, I don't know how to do it yet.
> 
> The quick solution would be to see if the device is active and if not fail
> the create request.
> 
> Vadim? What do you think?
Agree, failing a create request in this situation will be a good idea.
Best regards,
Vadim.
Comment 11 Ronen Hod 2012-12-06 13:11:00 UTC 
- Not required for WHQL
- S3/S4 support is not mandatory for RHEL6.4
Deferring to RHEL6.5
Comment 12 Dmitry Fleytman 2012-12-09 15:40:49 UTC 
Created attachment 660307 [details]
Fix for the bug based on private repository from git.engineering
Comment 13 Dmitry Fleytman 2012-12-09 15:41:54 UTC 
Fix for this bug based on Vadym's private repository is attached. Please, review it and commit as I don't have corresponding permissions.

Also pay attention that the patch doesn't fix all possible problems. See commit message for additional details.
Comment 14 Gal Hammer 2012-12-09 16:11:18 UTC 
Dmitry's patch seems to solve the crash. Thanks!

Although I'm not sure I fully understand this solution. For example, in the VIOSerialPortCreate function only the out and control queues are used. So why do you acquire a lock on the input buffer which is not used?
Comment 15 Dmitry Fleytman 2012-12-10 07:37:28 UTC 
It could be I missed something because I'm not deeply familiar with serial driver's code yet. I didn't change locks used in this function, I just ordered them in a proper way.

Also as I wrote in the commit message locking scheme of the driver needs to be redesigned as it doesn't look proper for me.
I believe separate bug needs to be open for this.
Comment 16 dawu 2013-01-18 08:31:50 UTC 
Reproduce this bug with virtio-win-prewhql-49 on win7-32,test this bug with virtio-win-prewhql-51 with steps as comment #0

Actually Results:
on version virtio-win-prewhql-49,qemu-kvm quit with "Guest move used index from 0 to 256"
on version virtio-win-prewhql-51, no BSOD happened, but after guest resume, virtio-serial function fail with error in qemu side :"(qemu) qemu-kvm: virtio-serial-bus: Guest failure in adding port 1 for device virtio-serial0.0".
virtio-serial can work well after reboot guest.

Based on above, this issue haven't been fixed yet, re-assign this bug.

Thanks!

Best Regards,
Dawn
Comment 17 Gal Hammer 2013-01-20 13:39:38 UTC 
(In reply to comment #16)
> Reproduce this bug with virtio-win-prewhql-49 on win7-32,test this bug with
> virtio-win-prewhql-51 with steps as comment #0
> 
> Actually Results:
> on version virtio-win-prewhql-49,qemu-kvm quit with "Guest move used index
> from 0 to 256"
> on version virtio-win-prewhql-51, no BSOD happened, but after guest resume,
> virtio-serial function fail with error in qemu side :"(qemu) qemu-kvm:
> virtio-serial-bus: Guest failure in adding port 1 for device
> virtio-serial0.0".
> virtio-serial can work well after reboot guest.
> 
> Based on above, this issue haven't been fixed yet, re-assign this bug.

Don't you think this is another issue which is not related to this one?
Comment 18 Mike Cao 2013-01-20 14:54:43 UTC 
(In reply to comment #17)
> (In reply to comment #16)
> > Reproduce this bug with virtio-win-prewhql-49 on win7-32,test this bug with
> > virtio-win-prewhql-51 with steps as comment #0
> > 
> > Actually Results:
> > on version virtio-win-prewhql-49,qemu-kvm quit with "Guest move used index
> > from 0 to 256"
> > on version virtio-win-prewhql-51, no BSOD happened, but after guest resume,
> > virtio-serial function fail with error in qemu side :"(qemu) qemu-kvm:
> > virtio-serial-bus: Guest failure in adding port 1 for device
> > virtio-serial0.0".
> > virtio-serial can work well after reboot guest.
> > 
> > Based on above, this issue haven't been fixed yet, re-assign this bug.
> 
> Don't you think this is another issue which is not related to this one?

Since no BOSD occurs ,we can consider the original issue has gone ,Dawu ,pls report a new bug to track your issue you hit .

Mike
Comment 19 dawu 2013-01-21 03:11:47 UTC 
Report another bug for the new issue, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=902153

Thanks 
Best Regards,
Dawn
Comment 20 Mike Cao 2013-01-21 03:15:04 UTC 
Since the Original issue has gone ,Move status to VERIFIED.
Need to mention that we decided to ship build49 to customers ,Pls keep product flags to RHEL6.5.0.

Mike
Comment 24 errata-xmlrpc 2013-11-22 00:00:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1729.html