Bug 989459

Summary: Windows guest stuck at bootng(using scsi-hd)
Product: Red Hat Enterprise Linux 7 Reporter: Xiaoqing Wei <xwei>
Component: qemu-kvmAssignee: Vadim Rozenfeld <vrozenfe>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0CC: hhuang, juzhang, michen, rbalakri, rhod, scui, shuang, sluo, virt-bugs, virt-maint, xwei
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virtio-win-prewhql-88 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-09 07:48:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
pic none

Description Xiaoqing Wei 2013-07-29 10:07:22 UTC
Description of problem:

Win2012r2 stuck at boot during os installation(using scsi-hd and aio=threads)

Version-Release number of selected component (if applicable):

kernel-3.10.0-2.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
virtio-win-prewhql-65

How reproducible:
only once, tried another 4 rounds, didn't manage to reproduce.

Steps to Reproduce:
1.
/root/staf-kvm-devel/autotest-devel/client/tests/virt/qemu/qemu \
    -S \
    -name 'virt-tests-vm1' \
    -nodefaults \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130726-143052-9Z2qISeN,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130726-143052-9Z2qISeN,server,nowait \
    -device isa-serial,chardev=serial_id_serial1 \
    -chardev socket,id=seabioslog_id_20130726-143052-9Z2qISeN,path=/tmp/seabios-20130726-143052-9Z2qISeN,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20130726-143052-9Z2qISeN,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,addr=0x5 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64.raw',if=none,id=virtio-scsi-id0,media=disk,cache=none,snapshot=off,format=raw,aio=threads \
    -device scsi-hd,drive=virtio-scsi-id0 \
    -device e1000,netdev=idODbbMw,mac='9a:0a:0b:0c:0d:0e',bus=pci.0,addr=0x3,id='idcKhjhx' \
    -netdev tap,id=idODbbMw,fd=22 \
    -m 4096 \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
    -cpu 'SandyBridge' \
    -M pc \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/ISO/Win2012R2/en_windows_server_2012_r2_datacenter_preview_x64_dvd_2358570.iso',if=none,id=virtio-scsi-id1,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi-id1 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso',if=none,id=virtio-scsi-id2,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi-id2 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/virtio-win.iso',if=none,id=virtio-scsi-id3,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi-id3 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64/answer.vfd',if=none,id=fdc0-0-0,media=disk,readonly=off,format=raw \
    -global isa-fdc.driveA=fdc0-0-0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -spice password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \
    -vga qxl \
    -global qxl-vga.vram_size=33554432 \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off  \
    -enable-kvm
2.
3.

Actual results:
Win2012r2 stuck at boot during os installation
last for at least 30mins before vm killed.
Expected results:
installation finished successfully, not stuck
Additional info:
same test work well when using aio=native




processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping	: 7
microcode	: 0x28
cpu MHz		: 1904.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 6784.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Comment 3 Xiaoqing Wei 2013-07-29 12:18:34 UTC
BTW, the guest os is not died when the stuck happened,

press the combination key: 'shift+F10' can open a cmd in guest.

Comment 4 Xiaoqing Wei 2013-08-07 02:39:11 UTC
met it again with 
qemu-kvm-1.5.2-2.el7.x86_64
3.10.0-3.el7.x86_64

Comment 5 Xiaoqing Wei 2013-08-07 08:54:12 UTC
This could also be triggered when using ide-cdrom(hard drive still scsi-hd)

/root/staf-kvm-devel/autotest-devel/client/tests/virt/qemu/qemu \
    -S \
    -name 'virt-tests-vm1' \
    -nodefaults \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130801-160234-9EJouOL8,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130801-160234-9EJouOL8,server,nowait \
    -device isa-serial,chardev=serial_id_serial1 \
    -chardev socket,id=seabioslog_id_20130801-160234-9EJouOL8,path=/tmp/seabios-20130801-160234-9EJouOL8,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20130801-160234-9EJouOL8,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,addr=0x5 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64.qcow2',if=none,id=virtio-scsi0-id0,media=disk,cache=none,snapshot=off,format=qcow2,aio=threads \
    -device scsi-hd,drive=virtio-scsi0-id0 \
    -device e1000,netdev=idb6MnUd,mac='9a:bb:bc:bd:be:bf',bus=pci.0,addr=0x3,id='idrud9iv' \
    -netdev tap,id=idb6MnUd,fd=22 \
    -m 4096 \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
    -cpu 'SandyBridge' \
    -M pc \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/ISO/Win2012R2/en_windows_server_2012_r2_datacenter_preview_x64_dvd_2358570.iso',index=1,if=none,id=drive-ide0-0-0,media=cdrom,format=raw \
    -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso',index=2,if=none,id=drive-ide0-0-1,media=cdrom,format=raw \
    -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/virtio-win.iso',index=3,if=none,id=drive-ide0-1-0,media=cdrom,format=raw \
    -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64/answer.vfd',if=none,id=fdc0-0-0,media=disk,readonly=off,format=raw \
    -global isa-fdc.driveA=fdc0-0-0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -vnc :0 \
    -vga std \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off  \
    -enable-kvm

Comment 6 Suqin Huang 2013-08-27 10:09:10 UTC
Hi xwei,
it only happen once, why do you think it's aio ans scsi-hd related.

if it's aio related can you provide the result with the following steps

I'm afraid the aio threads are not created by main thread.

1. boot guest with 4 vcpus.

2. stop guest

3. pin main thread to host cpu 1

4. pin one vcpu thread to one host cpu.

5. cont guest

get output with the following cmd:


import commands

thread_cmd = "ps -C qemu-kvm H -o spid="
thread_output = commands.getoutput(thread_cmd).split()

for i in range(len(thread_output)):                                             
    taskset_cmd = "taskset -p %s" % thread_output[i]
    taskset_cpu = commands.getoutput(taskset_cmd)
    print taskset_cpu

Comment 8 ShupingCui 2013-09-11 07:16:28 UTC
met it again with:
kernel-2.6.32-416.el6.x86_64
qemu-kvm-0.12.1.2-2.400.el6.x86_64
virtio-win-prewhql-0.1-69

stuck with virtio-scsi, and installation finished with virtio-blk.

Comment 9 ShupingCui 2013-09-11 10:10:20 UTC
command line:

/usr/libexec/qemu-kvm \
    -name 'virt-tests-vm1' \
    -nodefaults \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,addr=0x5 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012-64-virtio.raw',if=none,id=virtio-scsi0-id0,media=disk,cache=none,snapshot=off,format=raw,aio=threads \
    -device scsi-hd,drive=virtio-scsi0-id0 \
    -device virtio-net-pci,netdev=idvNJoj4,mac='9a:8e:8f:90:91:92',bus=pci.0,addr=0x3,id='idpuEowM' \
    -netdev tap,id=idvNJoj4,vhost=on,vhostfd=20,fd=18 \
    -m 8192 \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
    -cpu 'SandyBridge' \
    -M rhel6.5.0 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/ISO/Win2012/en_windows_server_2012_x64_dvd_915478.iso',if=none,id=virtio-scsi1-id1,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi1-id1 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso',if=none,id=virtio-scsi2-id2,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi2-id2 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/virtio-win-prewhql-0.1-69.iso',if=none,id=virtio-scsi3-id3,media=cdrom,readonly=on,format=raw \
    -device scsi-cd,drive=virtio-scsi3-id3 \
    -device AC97,addr=0x6 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012-64/answer.vfd',if=none,id=fdc0-0-0,media=disk,readonly=off,format=raw \
    -global isa-fdc.driveA=fdc0-0-0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -spice port=3000,password=123456,addr=0,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \
    -vga qxl \
    -global qxl-vga.vram_size=33554432 \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off  \
    -enable-kvm

Host cpuinfo:
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 58
model name	: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
stepping	: 9
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 6784.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Comment 10 Xiaoqing Wei 2013-09-11 11:22:40 UTC
Created attachment 796346 [details]
pic

this is happen when guest loading the virtio-scsi driver,
the progress bar is keep moving from left to right, and reset to left.
I observed this for at least 5 min

and on -69 prewhql, it can happend whatever native/threads.
now 100% on -69

and can happen on win2012 / win7 too, raise to urgent/urgent

Comment 11 Xiaoqing Wei 2013-12-17 11:59:49 UTC
Found that this could also happen on OS booting, 2012 R2 Server(with gui)

Comment 12 Ronen Hod 2014-01-02 10:28:34 UTC
QE,

Please retest with the latest virtio-win. There were relevant bug fixes in build 74.
Thanks.
Can you attach the unattended install (unattended.ini or something)

Comment 13 juzhang 2014-01-03 01:36:58 UTC
(In reply to Ronen Hod from comment #12)
> QE,
> 
> Please retest with the latest virtio-win. There were relevant bug fixes in
> build 74.
> Thanks.
> Can you attach the unattended install (unattended.ini or something)

Hi Xwei,

Could you have a try and update the result in bz?

Best Regards,
Junyi

Comment 14 Xiaoqing Wei 2014-01-03 02:43:13 UTC
Oh, Hello Ronen, Junyi,

Sorry for late response, the C#11 was using -74 virtio-win-prewhql
We shifted to it Since Nov 21, 2013 CST.

https://brewweb.devel.redhat.com/packageinfo?packageID=31338

Comment 17 Vadim Rozenfeld 2014-07-04 14:12:03 UTC
Any updates on this issue?
Is it still reproducible on rhel7.1 and the latest drivers?

Thanks,
Vadim.

Comment 18 Ronen Hod 2014-08-06 09:47:51 UTC
QE,
Since it doesn't reproduce on Vadim's setup,
Please test again with build-88
If it reproduces, Vadim will provide a debug version of the drivers in order to get a log through the serial port.

Comment 19 juzhang 2014-08-11 08:56:02 UTC
(In reply to Ronen Hod from comment #18)
> QE,
> Since it doesn't reproduce on Vadim's setup,
> Please test again with build-88
> If it reproduces, Vadim will provide a debug version of the drivers in order
> to get a log through the serial port.

Hi Scui,

Could you give a help for handling this?

Best Regards,
Junyi

Comment 20 ShupingCui 2014-09-03 07:21:26 UTC
Re-test 10 times with virtio-win-prewhql-88, cannot reproduce this bug.

Comment 21 Vadim Rozenfeld 2014-09-03 10:07:24 UTC
Thanks,
Can we close it?

Comment 23 ShupingCui 2014-09-08 13:58:42 UTC
(In reply to Vadim Rozenfeld from comment #21)
> Thanks,
> Can we close it?

Hi Vadim,

I think we can close this bug.

Thanks,
Shuping