Bug 1513833

Summary: whql job "Multiple processor group device test" fail when boot with cpu flag "hv_time/hv_relaxed/hv_vapic/hv_spinlocks=0x1fff "
Product: Red Hat Enterprise Linux 7 Reporter: Yu Wang <wyu>
Component: qemu-kvm-rhevAssignee: Amnon Ilan <ailan>
Status: CLOSED CURRENTRELEASE QA Contact: Yu Wang <wyu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: ailan, knoel, lijin, virt-maint, vrozenfe, wyu, xiagao
Target Milestone: rcKeywords: TestOnly
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-13 01:19:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1451959    
Bug Blocks:    
Attachments:
Description Flags
multi-processor-group-failed none

Description Yu Wang 2017-11-16 04:48:10 UTC 
Created attachment 1353234 [details]
multi-processor-group-failed

Description of problem:


Version-Release number of selected component (if applicable):
guest: win201264 and win2012r2
kernel-3.10.0-774.el7.x86_64
qemu-kvm-rhev-2.10.0-6.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot guest with 8 vcpus and hv cpu flags
/usr/libexec/qemu-kvm -name 143BLN201264CRD -enable-kvm -m 6G -smp 8 -uuid 0efb1199-5218-4bc2-8314-67060829b615 -nodefconfig -nodefaults -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -chardev socket,id=charmonitor,path=/tmp/143BLN201264CRD,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb -drive file=143BLN201264CRD,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=en_windows_server_2012_x64_dvd_915478.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=143BLN201264CRD.vfd,if=floppy,id=drive-fdc0-0-0,format=raw,cache=none -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:75:4d:f6:5e -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga std -M pc -device virtio-balloon-pci,id=balloon0,bus=pci.0


2.run job "Multiple processor group device test"

detailed steps:
1 boot guest

2 bcdedit.exe /set groupsize 1 
  bcdedit.exe /set maxgroup on 
  bcdedit.exe /set groupaware on 

3 reboot guest

4 Verify the system is configured into multiple active processor groups 
  Running groupsize.exe 


Actual results:
When boot guest with any fv flags below, the job will fail
"hv_time/hv_relaxed/hv_vapic/hv_spinlocks=0x1fff "

failed as "ValidateMultipleGroupsExist - Failure: Expected more than one active processor group, but found only one."

Expected results:
Pass

Additional info:
1 When boot without cpu flags , this will pass, but will hang at the first reboot (only win201264, win2012r2 not hang)
Bug 1235510 - [virtio-win][whql]WIn2012 guest could not boot up while running multiple processor job(reboot case)

2 When boot with cpu flags, it will not hang, but failed.(win201264)
Comment 2 Yu Wang 2017-11-16 06:14:05 UTC 
Tried on qemu-kvm-rhev-2.9.0-16.el7.x86_64, still hit this issue, so it is not a regression.

Thanks
Yu Wang
Comment 3 Vadim Rozenfeld 2017-11-17 00:54:57 UTC 
That is a quite interesting issue. The problem comes from the fact that before WS2012R2 Hyper-V didn't support more than 64 vCPUs which is exactly one group size. 
Please try build https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 from https://bugzilla.redhat.com/show_bug.cgi?id=1451959#c19  With the above changes we can run Windows VM with more than 64 VCPUs which means more than one processor group. (Just checked it with coreinfo utility available athttps://docs.microsoft.com/en-us/sysinternals/downloads/coreinfo. VM with 128 vCPUs has 2 groups).

Vadim.
Comment 4 Yu Wang 2017-11-17 03:20:44 UTC 
Hi Vadim,

(In reply to Vadim Rozenfeld from comment #3)
> That is a quite interesting issue. The problem comes from the fact that
> before WS2012R2 Hyper-V didn't support more than 64 vCPUs which is exactly
> one group size. 
> Please try build
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 from
> https://bugzilla.redhat.com/show_bug.cgi?id=1451959#c19  With the above
> changes we can run Windows VM with more than 64 VCPUs which means more than
> one processor group. (Just checked it with coreinfo utility available
> athttps://docs.microsoft.com/en-us/sysinternals/downloads/coreinfo. VM with
> 128 vCPUs has 2 groups).

Tried with this version above, the job can pass as before. 
re-tried with qemu-kvm-rhev-2.10.0-6.el7.x86_64, this job can filter pass with error in comment#0.

So if it can be counted as a bug now?

> 
> Vadim.
Comment 5 Vadim Rozenfeld 2017-11-17 03:53:05 UTC 
(In reply to Yu Wang from comment #4)
> Hi Vadim,
> 
> (In reply to Vadim Rozenfeld from comment #3)
> > That is a quite interesting issue. The problem comes from the fact that
> > before WS2012R2 Hyper-V didn't support more than 64 vCPUs which is exactly
> > one group size. 
> > Please try build
> > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 from
> > https://bugzilla.redhat.com/show_bug.cgi?id=1451959#c19  With the above
> > changes we can run Windows VM with more than 64 VCPUs which means more than
> > one processor group. (Just checked it with coreinfo utility available
> > athttps://docs.microsoft.com/en-us/sysinternals/downloads/coreinfo. VM with
> > 128 vCPUs has 2 groups).
> 
> Tried with this version above, the job can pass as before. 
> re-tried with qemu-kvm-rhev-2.10.0-6.el7.x86_64, this job can filter pass
> with error in comment#0.
>

So, the build from https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 solved the problem, right?
 
> So if it can be counted as a bug now?

I would say it was a feature or implementation limit, but since it is gone now, let's count it as a bug :)

Best,
Vadim.
 
> 
> > 
> > Vadim.
Comment 6 Yu Wang 2017-11-17 04:00:12 UTC 
Hi Vadim,
(In reply to Vadim Rozenfeld from comment #5)

> 
> So, the build from
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 solved
> the problem, right?

Yes, it solve the problem of "ValidateMultipleGroupsExist - Failure: Expected more than one active processor group, but found only one.", but still will hang at the first reboot on win2012, it is already reported before.

Bug 1235510 - [virtio-win][whql]WIn2012 guest could not boot up while running multiple processor job(reboot case)

Thanks
Yu Wang

>  
> > So if it can be counted as a bug now?
> 
> I would say it was a feature or implementation limit, but since it is gone
> now, let's count it as a bug :)
> 
> Best,
> Vadim.
>  
> > 
> > > 
> > > Vadim.
Comment 7 Vadim Rozenfeld 2017-11-17 04:15:30 UTC 
(In reply to Yu Wang from comment #6)
> Hi Vadim,
> (In reply to Vadim Rozenfeld from comment #5)
> 
> > 
> > So, the build from
> > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14535540 solved
> > the problem, right?
> 
> Yes, it solve the problem of "ValidateMultipleGroupsExist - Failure:
> Expected more than one active processor group, but found only one.", but
> still will hang at the first reboot on win2012, it is already reported
> before.
> 
> Bug 1235510 - [virtio-win][whql]WIn2012 guest could not boot up while
> running multiple processor job(reboot case)
> 

I see, does it happen when testing virtio-scsi-pci only? In any case could you please generate and share a new crash dump file ?

Thanks,
Vadim.

> Thanks
> Yu Wang
> 
> >  
> > > So if it can be counted as a bug now?
> > 
> > I would say it was a feature or implementation limit, but since it is gone
> > now, let's count it as a bug :)
> > 
> > Best,
> > Vadim.
> >  
> > > 
> > > > 
> > > > Vadim.
Comment 8 Yu Wang 2017-11-17 04:45:30 UTC 
(In reply to Vadim Rozenfeld from comment #7)

> > 
> > Yes, it solve the problem of "ValidateMultipleGroupsExist - Failure:
> > Expected more than one active processor group, but found only one.", but
> > still will hang at the first reboot on win2012, it is already reported
> > before.
> > 
> > Bug 1235510 - [virtio-win][whql]WIn2012 guest could not boot up while
> > running multiple processor job(reboot case)
> > 
> 
> I see, does it happen when testing virtio-scsi-pci only? In any case could
> you please generate and share a new crash dump file ?

No, it also happened on ide device, I will upload a new dump on bz1235510 for your further debug.

> 
> Thanks,
> Vadim.
>
Comment 9 Ladi Prosek 2017-11-21 11:59:51 UTC 
Based on comment 6 this is addressed with the same fix as bug 1451959 and the remaining issue is already being tracked with bug 1235510.

Adding a dependency on bug 1451959 and the TestOnly keyword.
Comment 10 Yu Wang 2017-11-28 07:37:58 UTC 
verified this bug with qemu-kvm-rhev-2.10.0-8.el7.x86_64

steps as comment#0, remaining issue is already being tracked with bug 1235510.

So this bug has been fixed, change to verified status.

Thanks
Yu Wang