Bug 1022895

Created attachment 815703 [details]
all debug file

I have attached all debug files which are generated by autotest, there is guest screendump, debug info and some system info. 

If there is any problem, please let me know, thanks.

met this problem on
kernel-3.10.0-61.el7.x86_64
qemu-kvm-1.5.3-21.el7.x86_64

How reproducible:
3/11

Host cpu info:
processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Xeon(R) CPU           E5205  @ 1.86GHz
stepping	: 6
microcode	: 0x60f
cpu MHz		: 1861.944
cache size	: 6144 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips	: 3723.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 38 bits physical, 48 bits virtual
power management:

Hit this bug w/ win2012r2 guest on version:

kernel-3.10.0-65.el7.x86_64
qemu-kvm-rhev-1.5.3-31.el7.x86_64
virtio-win-prewhql-0.1-74.iso

will attach the debug file.

Created attachment 846921 [details]
all debug file - win2012r2 guest

Is this hang reproducible without AHCI?

Best regards,
Yan.

(In reply to Yan Vugenfirer from comment #7)
> Is this hang reproducible without AHCI?
> 
> Best regards,
> Yan.

Yes, it can be reproduced without AHCI.
In comment 5, win2012r2 guest used virtio_scsi instead of AHCI.

-drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/home/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64-virtio.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \ 

And I have attached the debug file which has the screendumps in comment 6.

If it is necessary, I will test Win8.1.i386 guest.

Thanks,
Cong

Can you generate guest crash dump when the hang is happening?

Thanks,
Yan.

Mike,

Can you please try to reproduce, and isolate the cause for this issue.
I think that you dis not run into this issue.

Thanks, Ronen.

(In reply to Ronen Hod from comment #10)
> Mike,
> 
> Can ou please try to reproduce, and isolate the cause for this issue.
> I think that you dis not run into this issue.
> 
> Thanks, Ronen.

I installed win8.1-32 guest without any virtio-win drivers for more than 10 times,did not hit any issue,but we find some hang issues during netkvm whql test even without e1000 driver.I will try virtio-scsi tomorrow

(In reply to Yan Vugenfirer from comment #9)
> Can you generate guest crash dump when the hang is happening?
> 
> Thanks,
> Yan.

Hi Yan,

Can't get the dump file for the installation just didn't start, the guest doesn't have OS.

can hit this bug with win2012r2 and driver virtio-blk & virtio-net.

    -m 4096
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=threads,file=/home/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win2012r2-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=06 \
    -device virtio-net-pci,mac=9a:26:27:28:29:2a,id=idm9SL2r,netdev=id6u6Dn3,bus=pci.0,addr=07  \

# free -m
             total       used       free     shared    buffers     cached
Mem:         47897      21083      26814         12          1      16212
-/+ buffers/cache:       4869      43028
Swap:        24135          0      24135

Tested on version:
kernel-3.10.0-84.el7.x86_64
qemu-kvm-1.5.3-45.el7.x86_64
virtio-win-prewhql-0.1-74.iso

The cdrom(drive_cd1) rd_bytes just same.

10 mins ago:
(qemu) info blockstats 
drive_image1: rd_bytes=3072 wr_bytes=512 rd_operations=6 wr_operations=1 flush_operations=0 wr_total_time_ns=108504794 rd_total_time_ns=285716 flush_total_time_ns=0
drive_cd1: rd_bytes=221749248 wr_bytes=0 rd_operations=108276 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=2105619769 flush_total_time_ns=0
drive_winutils: rd_bytes=2048 wr_bytes=0 rd_operations=1 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=99709 flush_total_time_ns=0
drive_virtio: rd_bytes=2048 wr_bytes=0 rd_operations=1 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=97656 flush_total_time_ns=0
drive_fl: rd_bytes=0 wr_bytes=0 rd_operations=0 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=0 flush_total_time_ns=0

10 mins later:
(qemu) info blockstats 
drive_image1: rd_bytes=3072 wr_bytes=512 rd_operations=6 wr_operations=1 flush_operations=0 wr_total_time_ns=108504794 rd_total_time_ns=285716 flush_total_time_ns=0
drive_cd1: rd_bytes=221749248 wr_bytes=0 rd_operations=108276 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=2105619769 flush_total_time_ns=0
drive_winutils: rd_bytes=2048 wr_bytes=0 rd_operations=1 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=99709 flush_total_time_ns=0
drive_virtio: rd_bytes=2048 wr_bytes=0 rd_operations=1 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=97656 flush_total_time_ns=0
drive_fl: rd_bytes=0 wr_bytes=0 rd_operations=0 wr_operations=0 flush_operations=0 wr_total_time_ns=0 rd_total_time_ns=0 flush_total_time_ns=0

Will attach the CML of win2012r2.

If there is other info I can collect, feel free to tell me.

Thanks,
Cong

Created attachment 861685 [details]
win2012r2 CML (virtio_blk and virtio_net)

Reproduced on 7.1 host and qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 when installed win2012r2 guest with virtio_scsi + e1000 + '-M rhel6.4.0'.

cmd line:
/usr/bin/qemu-kvm \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -M rhel6.4.0  \
    -nodefaults  \
    -vga std \
    -device AC97,bus=pci.0,addr=03  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20141205-181541-JxBhINvj,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20141205-181541-JxBhINvj,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20141205-181541-JxBhINvj,path=/tmp/seabios-20141205-181541-JxBhINvj,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20141205-181541-JxBhINvj,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=05 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/mnt/nfs_images/win2012-64r2-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device e1000,mac=9a:f9:72:56:c3:83,id=idCguBvR,netdev=idOjY14q,bus=pci.0,addr=06  \
    -netdev tap,id=idOjY14q,fd=24  \
    -m 16384  \
    -smp 21,maxcpus=21,cores=5,threads=2,sockets=2  \
    -cpu 'SandyBridge',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Win2012R2/en_windows_server_2012_r2_x64_dvd_2707946.iso \
    -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -drive id=drive_winutils,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/local/autotest/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-cd,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 \
    -drive id=drive_unattended,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/local/autotest/tests/virt/shared/data/images/win2012r2-64/autounattend.iso \
    -device ide-cd,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off \
    -enable-kvm

Created attachment 965686 [details]
screenshot

"-M pc" not met this bug, seems met it with "-M rhel6.4.0" now.

Hi Yan,

Could you have a look at this bug?

Thanks.

Hi,

Can you share please your autounattend.iso that was used to reproduce the bug?

Thanks.

Created attachment 1011109 [details]
Boot loader error 0x0000000A for -M rhel6.4.0

*** Bug 1209039 has been marked as a duplicate of this bug. ***

Currently reproducible with:

Command line:

/usr/libexec/qemu-kvm \
        -M pc \
        -nodefaults  \
        -vga std \
        -cpu 'SandyBridge',+sep \
        -enable-kvm \
        -m 4096\
        -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
        -name RH-1022895 \
        -rtc base=localtime,clock=host,driftfix=slew \
        -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x4 \
        -drive file=/home/igord/rh/1022895/virtio-scsi.qcow2 \
        -netdev tap,id=hostnet0,script=/home/igord/vms/whql-ifup-e1000,ifname=e1000-net0 \
        -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:51:51:51,bus=pci.0,addr=03 \
        \
        -boot order=cdn,once=d,menu=off \
        -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/igord/rh/1022895/en_windows_8_1_enterprise_x86_dvd_2972289.iso \
        -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.1,unit=0 \
        -drive id=drive_cd2,if=none,snapshot=off,aio=native,media=cdrom,file=/home/igord/rh/1022895/virtio-win-0.1-74.iso \
        -device ide-cd,id=cd2,drive=drive_cd2,bus=ide.1,unit=1 \
        \
        -k en-us \
        -vga std \
        \
        -chardev socket,id=abcde,path=/tmp/monitor-1022895,server,nowait \
        -mon chardev=abcde,mode=readline \
        \
        -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
        -vnc :0


Version-Release number of selected component (if applicable):
kernel.x86_64: 3.10.0-123.el7
qemu-kvm.x86_64: 10:1.5.3-60.el7

Currently reproducible with:

Command line:

/usr/libexec/qemu-kvm \
        -M pc \
        -nodefaults  \
        -vga std \
        -cpu 'SandyBridge',+sep \
        -enable-kvm \
        -m 4096\
        -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
        -name RH-1022895 \
        -rtc base=localtime,clock=host,driftfix=slew \
        -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x4 \
        -drive file=/home/igord/rh/1022895/virtio-scsi.qcow2 \
        -netdev tap,id=hostnet0,script=/home/igord/vms/whql-ifup-e1000,ifname=e1000-net0 \
        -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:51:51:51,bus=pci.0,addr=03 \
        \
        -boot order=cdn,once=d,menu=off \
        -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/igord/rh/1022895/en_windows_8_1_enterprise_x86_dvd_2972289.iso \
        -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.1,unit=0 \
        -drive id=drive_cd2,if=none,snapshot=off,aio=native,media=cdrom,file=/home/igord/rh/1022895/virtio-win-0.1-74.iso \
        -device ide-cd,id=cd2,drive=drive_cd2,bus=ide.1,unit=1 \
        \
        -k en-us \
        -vga std \
        \
        -chardev socket,id=abcde,path=/tmp/monitor-1022895,server,nowait \
        -mon chardev=abcde,mode=readline \
        \
        -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
        -vnc :0


Version-Release number of selected component (if applicable):
kernel.x86_64: 3.10.0-123.el7
qemu-kvm.x86_64: 10:1.5.3-60.el7

Doesn't reproduce here.

Can anyone who has a machine where it happens please enable all usb_xhci_* tracepoints and attach the log?

Also: there have been a number of xhci fixes for 2.9,
please retest once 2.9 builds are available.

(In reply to Gerd Hoffmann from comment #55)
> Doesn't reproduce here.
> 
> Can anyone who has a machine where it happens please enable all usb_xhci_*
> tracepoints and attach the log?

(In reply to Gerd Hoffmann from comment #56)
> Also: there have been a number of xhci fixes for 2.9,
> please retest once 2.9 builds are available.

Please retest (setting the right needinfo now)

Tested on:
qemu-kvm-rhev-2.9.0-3.el7.x86_64


1. There is no previous stuck issue now.

2. But met that Win8.1.32 could not apply the unattend answer's file's problem in windows setup with virtio-win-prewhql-0_1-136_i386.iso , screenshot will be attached.


Host info:
processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 1573.515
cache size	: 15360 KB
physical id	: 1
siblings	: 12
core id		: 5
cpu cores	: 6
apicid		: 43
initial apicid	: 43
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
bogomips	: 4004.34
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

Thanks.

Created attachment 1277497 [details]
screenshot - Win8.1.i386 could not apply unattend answer file - QEMU 2.9

(In reply to CongLi from comment #59)
> Created attachment 1277497 [details]
> screenshot - Win8.1.i386 could not apply unattend answer file - QEMU 2.9

Are you using same VM configuration or switched to virtio-block or virtio-scsi?

(In reply to Yan Vugenfirer from comment #60)
> (In reply to CongLi from comment #59)
> > Created attachment 1277497 [details]
> > screenshot - Win8.1.i386 could not apply unattend answer file - QEMU 2.9
> 
> Are you using same VM configuration or switched to virtio-block or
> virtio-scsi?


1. The hang problem was gone with previous configurations using ide.

2. For virtio_blk, it did not hang either before entering the installer, but there was the problem in attachment.


Thanks.

Do you use same answer file?

I think for virtio-block you need separate answer file, because now the storage driver is no longer Windows inbox driver.

(In reply to Yan Vugenfirer from comment #62)
> Do you use same answer file?
>
> I think for virtio-block you need separate answer file, because now the
> storage driver is no longer Windows inbox driver.

Thanks Yan for pointing it out, I did not describe it clearly.

For ide, using the same autoattend.iso as comment 51 but can not reproduce the previous issue now.

For virtio_blk, I used a different answer file for testing.


Host info:
processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 2500.078
cache size	: 15360 KB
physical id	: 1
siblings	: 12
core id		: 5
cpu cores	: 6
apicid		: 43
initial apicid	: 43
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts
bogomips	: 4004.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:


Thanks.

We had better update the flag from rhel-7.4.0 to rhel-7.5.0 if we do not plan to fix it in RHEL7.4

(In reply to CongLi from comment #63)
> (In reply to Yan Vugenfirer from comment #62)
> > Do you use same answer file?
> >
> > I think for virtio-block you need separate answer file, because now the
> > storage driver is no longer Windows inbox driver.
> 
> Thanks Yan for pointing it out, I did not describe it clearly.
> 
> For ide, using the same autoattend.iso as comment 51 but can not reproduce
> the previous issue now.
> 
> For virtio_blk, I used a different answer file for testing.
> 
> 
> Host info:
> processor	: 23
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 45
> model name	: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
> stepping	: 7
> microcode	: 0x710
> cpu MHz		: 2500.078
> cache size	: 15360 KB
> physical id	: 1
> siblings	: 12
> core id		: 5
> cpu cores	: 6
> apicid		: 43
> initial apicid	: 43
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 13
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
> pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb
> rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
> nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx
> est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
> tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority
> ept vpid xsaveopt dtherm ida arat pln pts
> bogomips	: 4004.81
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 46 bits physical, 48 bits virtual
> power management:
> 
> 
> Thanks.

Just to clarify the current status - so the issue is gone and the problem with answer file is fixed, I am right?

(In reply to Yan Vugenfirer from comment #65)
> 
> Just to clarify the current status - so the issue is gone and the problem
> with answer file is fixed, I am right?

Yes, according to my test, there is no issue now.

Thanks.