Bug 771390

Created attachment 550449 [details]
libvirt log file from crash

Created attachment 550450 [details]
/proc/cpuinfo from host

Created attachment 550451 [details]
dmesg from host

Created attachment 550452 [details]
dmidecode output from host

qemu command line is:

/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 3584 -smp 2,sockets=2,cores=1,threads=1 -name win7 -uuid afae711f-6754-3963-4892-e2348e4b9ae6 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -drive if=none,id=drive-fdc0-0-0,format=raw -global isa-fdc.driveA=drive-fdc0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/dev/root_vg/win7_vm_lv,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f9:b9:2f,bus=pci.0,addr=0x3 -usb -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

I set up a 64-bit guest, and it ran overnight without crashing, so this *appears* to only affect 32-bit guests.

I just reproduced the crash on my ThinkPad T510, with a pre-SandyBridge Core i7 M620, so it does not appear to be SandyBridge-specific.

(In reply to comment #8)
> I just reproduced the crash on my ThinkPad T510, with a pre-SandyBridge Core i7
> M620, so it does not appear to be SandyBridge-specific.

Hi, Ian

Could you give me more info , I tried following steps ,can not reproduce this issue.

1.boot a win7 32 bit (with virtio-blk, virito-nic virtio-serial driver)
 /usr/libexec/qemu-kvm -m 1G -smp 2 -usbdevice tablet -drive file=/home/win7_32,if=none,id=drive-ide0-0-0,cache=off,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,addr=0x9,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,mac=76:0A:49:3F:2A:25,bus=pci.0,addr=0x4,id=net0 -boot c -uuid a2bfcadf-ebf5-1429-3744-5a6f788b90a2 -rtc-td-hack -monitor stdio -name SPICE-W7 -enable-kvm -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=4,bus=pci.0 -chardev socket,server,host=0.0.0.0,port=12345,id=channel0,nowait -device virtserialport,chardev=channel0,nr=1,id=serialport.1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0 -chardev socket,server,host=0.0.0.0,port=12346,id=channel1,nowait -device virtserialport,chardev=channel1,nr=2,id=serialport.2,name=com.redhat.spice.0,bus=virtio-serial0.0 -spice port=5910,disable-ticketing,password -device virtio-balloon-pci,id=balloon0
2.begin streaming audio from http://theticket.com
3.begin streaming audio from youtube.

Actual Results:
guest works fine for 7 hours ,the guest did not quit with "virtio: trying to map MMIO memory"

(In reply to comment #9)
> Actual Results:
> guest works fine for 7 hours ,the guest did not quit with "virtio: trying to
> map MMIO memory"

There are a few things that may be at play here:

1)  The amount of RAM.  I gave the guest 3584MB in virt-manager.

2)  Intel VT-d.  I turned off VT-d in the BIOS of my laptop before going to
    lunch, and the guest has been running for more than an hour with no
    crash.  (It normally only takes a few minutes.)

3)  SPICE.  Did you install the SPICE QXL driver in the guest?  One of my
    co-workers in the Dallas office tried to replicate the problem and his
    guest didn't crash over lunch.  He had not installed the SPICE driver,
    and his guest was only 1 GB.  Not sure what made the difference.

Please continue to bisec the issue, my hunch is that it may be related to spice or to this exact memory size. In addition, its weird that the message of virtio complains about mmio since we only use port io in virtio...

Can you specify exact QEMU version (commit hash\repository if possible) that is used when the problem reproduced. 

I think at least we should look at the area of the printout from QEMU.

Yan told me offline it's a Fedora16 bug, doesn't affect RHEL6, so I'm clearing the flags.

It should be moved to RHEL7, but there's not virtio-win component there (why?)

In line 15 of https://bugzilla.redhat.com/show_bug.cgi?id=771390#c0
it says that it is a RHEL6.2 bug too.
Devel_ack+ for 6.3 for now.

Waiting for Mike from QE and Ian to figure out how it is reproducible.

(In reply to comment #15)
> In line 15 of https://bugzilla.redhat.com/show_bug.cgi?id=771390#c0
> it says that it is a RHEL6.2 bug too.
> Devel_ack+ for 6.3 for now.
> 
> Waiting for Mike from QE and Ian to figure out how it is reproducible.

I modified my configuration (install qxl-win ,extend memory to 3G),can *not* reproduce this issue . 
My guest works fine for 3 days (more than 72 hours ).

Ian ,I also notice that you were saying it happened on rhel6 ,Could you supply me the version of qemu-kvm ,qxl-win ?

Best Regards,
Mike

(In reply to comment #16)
> I modified my configuration (install qxl-win ,extend memory to 3G),can *not*
> reproduce this issue . 
> My guest works fine for 3 days (more than 72 hours ).

I am also having trouble reproducing this on a newly created guest, so I've been trying to figure out what the critical factor is that makes my guest crash.

At this point, it looks like Avast Antivirus is part of the equation uninstalling it makes the guest stable, and reinstalling it causes the guest to go back to crashing.

I'm going to (once again) try to reproduce this on a brand new guest and will report back.

> Ian ,I also notice that you were saying it happened on rhel6 ,Could you supply
> me the version of qemu-kvm ,qxl-win ?

I've seen this crash on Fedora 15 (qemu-system-x86-0.14.0-8.fc15.x86_64), Fedora 16 (qemu-system-x86-0.15.1-3.fc16.x86_64), and RHEL 6.2 (qemu-kvm-0.12.1.2-2.209.el6_2.1.x86_64).  It's the same guest image in all cases; the QXL driver is version 4.5.52942.0 (dated 5/16/2011).

I have just reproduced the crash on a new guest with Avast Antivirus installed.

Naive Windows guy question - if the guest is crashing (QEMU process), can't we get core dump for the process? It will so much easier to just look in the debugger at the point of crash.

(In reply to comment #20)
> Naive Windows guy question - if the guest is crashing (QEMU process), can't we
> get core dump for the process? It will so much easier to just look in the
> debugger at the point of crash.

I think the bug summary should use "quit" instead of "crash " we have lots of bugs about guest quit with "virtio: trying to map MMIO memory" .none of them has vmcore file .
eg :
https://bugzilla.redhat.com/show_bug.cgi?id=720540
https://bugzilla.redhat.com/show_bug.cgi?id=727034
https://bugzilla.redhat.com/show_bug.cgi?id=612801

(In reply to comment #18)
> I have just reproduced the crash on a new guest with Avast Antivirus installed.

Thanks.Reproduced this issue After install Avast Antivirus .Marked qa_ack.

This bug is related to virtio-blk and anti-virus. Guest will quit immediately when Avast Antivius on scaning . 

I will narrow down which virtio-blk version this bug introduced .

(In reply to comment #23)
> This bug is related to virtio-blk and anti-virus. Guest will quit immediately
> when Avast Antivius on scaning . 
> 
> I will narrow down which virtio-blk version this bug introduced .
This is not a regression bug .the bug exists on virtio-win-1.2.0.el6 virtio-win-1.3.0.1.el6 as well.

(In reply to comment #24)
> This is not a regression bug .the bug exists on virtio-win-1.2.0.el6
> virtio-win-1.3.0.1.el6 as well.

I never saw the problem when using the VirtIO drivers from RHEV 2.2.

(In reply to comment #21)
> I think the bug summary should use "quit" instead of "crash " we have lots of
> bugs about guest quit with "virtio: trying to map MMIO memory" .none of them
> has vmcore file .

This seems rather silly.  Should I file an RFE to treat VirtIO problems as crashes?

(In reply to comment #25)
> (In reply to comment #24)
> > This is not a regression bug .the bug exists on virtio-win-1.2.0.el6
> > virtio-win-1.3.0.1.el6 as well.
> 
> I never saw the problem when using the VirtIO drivers from RHEV 2.2.

RHEV2.2 was using rhel5 virtio-win(virtio-win-XXX.noarch.el5.rpm)
drivers instead of RHEL6 virtio-win drivers (virtio-win-XXX.noarch.el6.rpm)

(In reply to comment #27)
> RHEV2.2 was using rhel5 virtio-win(virtio-win-XXX.noarch.el5.rpm)
> drivers instead of RHEL6 virtio-win drivers (virtio-win-XXX.noarch.el6.rpm)

I was just trying to clarify that there is a version of the VirtIO drivers (albeit a relatively ancient version) that doesn't exhibit this problem -- which might be useful for your analysis.

Hi Mike,
Could you try our latest viostor driver from http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/20/win/virtio-win-prewhql-0.1.zip

Thank you,
Vadim.

(In reply to comment #29)
> Hi Mike,
> Could you try our latest viostor driver from
> http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/20/win/virtio-win-prewhql-0.1.zip

Hi, Vadim

Tried .Still hit this issue .

Best Regards,
Mike 

> 
> Thank you,
> Vadim.

FWIW, I managed to use gdb to produce a core dump of a "quitting" guest.  It's available at http://beanie.usersys.redhat.com/core.19029.gz (on the Red Hat internal network).

The steps I took to produce the core dump were:

1. Start the guest and immediately pause it.
2. In a shell, use ps to identify the PID of the guest.
3. Start gdb and attach to the process.  (Run through however many debuginfo-
   install cycles are required.)
4. Set a breakpoint at virtio.c:411.
5. Continue the process.
6. Unpause the guest and take the steps necessary to reproduce the VirtIO error.
7. When the error occurs, use gdb's generate-core-file command to create a
   core dump.

Thank you guys,
Manage to reproduce and hopefully fix the problem.

Without going too deep, the problem happened because
Avast Antivius bypasses file system and tends to work with
a very big chunks of data (much bigger than viostor driver 
can handle at the moment). The only reasonable solution
for this problem will be to clip the data transfer size and
report error in case of overrun.

Should be fixed in the recent build
http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/21/win/virtio-win-prewhql-0.1.zip

Best,
Vadim.

(In reply to comment #32)
> Thank you guys,
> Manage to reproduce and hopefully fix the problem.
> 
> Without going too deep, the problem happened because
> Avast Antivius bypasses file system and tends to work with
> a very big chunks of data (much bigger than viostor driver 
> can handle at the moment). The only reasonable solution
> for this problem will be to clip the data transfer size and
> report error in case of overrun.

Does it mean that Avast will not work properly in a guest? Maybe it will not solve the user's problem.
Is there a way to publish a limited max data size (or whatever).

(In reply to comment #35)
> (In reply to comment #32)
> > Thank you guys,
> > Manage to reproduce and hopefully fix the problem.
> > 
> > Without going too deep, the problem happened because
> > Avast Antivius bypasses file system and tends to work with
> > a very big chunks of data (much bigger than viostor driver 
> > can handle at the moment). The only reasonable solution
> > for this problem will be to clip the data transfer size and
> > report error in case of overrun.
> 
> Does it mean that Avast will not work properly in a guest? Maybe it will not
It will. With the latest fix it will be able to split one large chunk of data into several small requests. 
> solve the user's problem.
> Is there a way to publish a limited max data size (or whatever).
It what we do, but maximum transfer size is a contract between disk and file system subsystems, not between disk and someone who sends requests directly to
disk, bypassing file system. (btw, it is a pretty logical behaviour for anti-virus software)

New drivers seem stable in initial testing.  Avast virus scan appeared to complete successfully as well.

(In reply to comment #37)
> New drivers seem stable in initial testing.  Avast virus scan appeared to
> complete successfully as well.

Good news,
Thank you.

Mike, can we also try Avast on XP/W2K3?

Best,
Vadim.

(In reply to comment #38)
> (In reply to comment #37)
> > New drivers seem stable in initial testing.  Avast virus scan appeared to
> > complete successfully as well.
> 
> Good news,
> Thank you.
> 
> Mike, can we also try Avast on XP/W2K3?
> 
> Best,
> Vadim.

According to the previous comment 

winxp_32 ---> still exists
win2k3_32 ----> can not install
win7_32 -----> fixed

Hi, Vadim
1.Need QE test 64bit Guest ?
2.How to deal with this bug ,re-assign or open a new bug to debug winxp_32bit guest ?

Best Regards,
Mike

(In reply to comment #40)
> (In reply to comment #38)
> > (In reply to comment #37)
> > > New drivers seem stable in initial testing.  Avast virus scan appeared to
> > > complete successfully as well.
> > 
> > Good news,
> > Thank you.
> > 
> > Mike, can we also try Avast on XP/W2K3?
> > 
> > Best,
> > Vadim.
> 
> According to the previous comment 
> 
> winxp_32 ---> still exists
> win2k3_32 ----> can not install
> win7_32 -----> fixed
> 
> Hi, Vadim
> 1.Need QE test 64bit Guest ?
> 2.How to deal with this bug ,re-assign or open a new bug to debug winxp_32bit
> guest ?
> 
> Best Regards,
> Mike

Hi Mike,

I think it will be better to re-assign it back to me.
I will make a new build in the following days, so we 
should be able to validate this problem on all supported 
platforms.
best regards,
Vadim.

based on comment #41 ,re-assign this bug .

Hi Mike,
Could you try the new XP driver from http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip ?

Thank you,
Vadim.

(In reply to comment #43)
> Hi Mike,
> Could you try the new XP driver from
> http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
> ?
> 
> Thank you,
> Vadim.

Hi, Vadim

Tried with virito-win-prewhql-22 with win XP guest .

steps same as comment #0

Actual Results
Guest works find after anti-virus scanning .

Based on above ,the driver fixed the issue for winxp .

QE will check whether 64 bit guest works or not

Hi all,

   According to comments 39,I tried the bug on windows xp 32/windows7 64bits guest with with virito-win-prewhql-22.The scanning of anti-virus successfully finished at last and the guest works well without any error info.
  so the bug should be fixed.

Thanks 
Min

(In reply to comment #45)
> Hi all,
> 
>    According to comments 39,I tried the bug on windows xp 32/windows7 64bits
> guest with with virito-win-prewhql-22.The scanning of anti-virus successfully
> finished at last and the guest works well without any error info.
>   so the bug should be fixed.
> 
> Thanks 
> Min

Thank you, Min.
Could you move it to verified?

Best regards,
Vadim.

I am unable to resolve the URL from Comment #44:

rick@rick ~ $ wget http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
--2012-02-16 10:11:21--  http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
Resolving download.devel.redhat.com... failed: Name or service not known.
wget: unable to resolve host address `download.devel.redhat.com'

Is there some other way to obtain the driver?
Thanks,
-Rick

(In reply to comment #47)
> I am unable to resolve the URL from Comment #44:
> 
> rick@rick ~ $ wget
> http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
> --2012-02-16 10:11:21-- 
> http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
> Resolving download.devel.redhat.com... failed: Name or service not known.
> wget: unable to resolve host address `download.devel.redhat.com'
> 
> Is there some other way to obtain the driver?
> Thanks,
> -Rick


http://people.redhat.com/vrozenfe/vioscsi.vfd
But don't forget, they are not WHQL'ed yet.
So, use them at your own risk.
Vadim.

(In reply to comment #48)
> (In reply to comment #47)
> > I am unable to resolve the URL from Comment #44:
> > 
> > rick@rick ~ $ wget
> > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
> > --2012-02-16 10:11:21-- 
> > http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/22/win/virtio-win-prewhql-0.1.zip
> > Resolving download.devel.redhat.com... failed: Name or service not known.
> > wget: unable to resolve host address `download.devel.redhat.com'
> > 
> > Is there some other way to obtain the driver?
> > Thanks,
> > -Rick
> 
> 
> http://people.redhat.com/vrozenfe/vioscsi.vfd
> But don't forget, they are not WHQL'ed yet.
> So, use them at your own risk.
> Vadim.

Understood. Thank you *very* much.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:  viostor utility wasn't checking incoming buffer size
Consequence: applications can send buffers much bigger than maximum transfer size to viostor driver directly, by bypassing file system stack.
Fix: trim buffer size if it is bigger than maximum transfer size. 
Result: viostor driver can properly handle requests with buffers of any size.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0751.html