Bug 1628965
| Summary: | Performance Degradation running PVP on all packet sizes. rfc 2544 Throughput tests | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Bob Fubel <bfubel> | ||||
| Component: | qemu-kvm-rhev | Assignee: | Maxime Coquelin <maxime.coquelin> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Pei Zhang <pezhang> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.6 | CC: | ailan, atragler, chayang, ctrautma, juzhang, maxime.coquelin, michen, pezhang, siliu, tli, virt-maint | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-09-18 14:14:45 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Bob Fubel
2018-09-14 14:16:11 UTC
Created attachment 1483343 [details]
Xena Test File
I and Ting are debugging this lower throughput issue, we found probably the vCPU ping cause it. As the interface of qemu 'info cpus' changes a bit like below, this may affect the automation script to set vCPU ping.
With qemu 2.10:
(qemu) info cpus
* CPU #0: pc=0x0000000007f06173 (halted) thread_id=11945
CPU #1: pc=0x00000000000fd0f5 (halted) thread_id=11946
CPU #2: pc=0x00000000000fd0f5 (halted) thread_id=11947
With qemu 2.12:
(qemu) info cpus
* CPU #0: thread_id=22877
CPU #1: thread_id=22878
CPU #2: thread_id=22879
CPU #3: thread_id=22880
CPU #4: thread_id=22881
By the way, we don't hit this throughput issue, it always works as expected with both qemu2.10 and qemu2.12. We are testing from libvirt layer, that's why we don't affected by above "info cpus" changes. Besides, the emulator CPUs don't share same host CPUs with vCPUs. Below is the libvirt CPU configuration which we use.
<vcpu placement='static'>6</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='30'/>
<vcpupin vcpu='1' cpuset='31'/>
<vcpupin vcpu='2' cpuset='29'/>
<vcpupin vcpu='3' cpuset='27'/>
<vcpupin vcpu='4' cpuset='25'/>
<vcpupin vcpu='5' cpuset='23'/>
<emulatorpin cpuset='18,20,22,24,26,28'/>
</cputune>
*** Bug 1616306 has been marked as a duplicate of this bug. *** Hi Pei, In your setup, are you trying QEMU 2.12 on RHEL 7.5 or RHEL 7.6? Li Ting is testing tonight to confim the new way of pinning the CPU's
will work for all the longer tests. if it does then we will be able to close off this bug. And document the change in the way the pinning is working.
Bob
<vcpu placement='static'>6</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='30'/>
<vcpupin vcpu='1' cpuset='31'/>
<vcpupin vcpu='2' cpuset='29'/>
<vcpupin vcpu='3' cpuset='27'/>
<vcpupin vcpu='4' cpuset='25'/>
<vcpupin vcpu='5' cpuset='23'/>
<emulatorpin cpuset='18,20,22,24,26,28'/>
</cputune>
On 09/15/2018 10:13 AM, Amnon Ilan wrote:
> Hi Chris, Bob,
>
> Please note that you cannot change the qemu version without the RHEL version.
> RHEL7.6 goes with qemu-2.12
> RHEL7.5 goes with qemu-2.10
>
> That's the way virt-qe are testing it, so if you want to switch to qemu-2.12 you should
> also switch to RHEL7.6.
>
> From the new BZ is seems that you are trying an untested combination: 7.5+qemu-2.12
>
> Thanks,
> Amnon
>
>
>
> On Fri, Sep 14, 2018 at 7:57 PM Christian Trautman <ctrautma> wrote:
>
> Just to clarify, the performance issue is only when updating to qemu 2.12.
>
> All tests with 7.5 with qemu 2.10 have not had a problem with either ovs 2.9 or ovs 2.10.
>
> We closed the previous bug and opened a new one to get the information precise and easier to follow.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1628965
>
> -Christian,
(In reply to Maxime Coquelin from comment #6) > Hi Pei, > > In your setup, are you trying QEMU 2.12 on RHEL 7.5 or RHEL 7.6? Hi Maxime, I'm testing QEMU 2.12 on RHEL7.6. Best regards, Pei (In reply to Bob Fubel from comment #7) > Li Ting is testing tonight to confim the new way of pinning the CPU's > will work for all the longer tests. if it does then we will be able to close > off this bug. And document the change in the way the pinning is working. > > Bob > > <vcpu placement='static'>6</vcpu> > <cputune> > <vcpupin vcpu='0' cpuset='30'/> > <vcpupin vcpu='1' cpuset='31'/> > <vcpupin vcpu='2' cpuset='29'/> > <vcpupin vcpu='3' cpuset='27'/> > <vcpupin vcpu='4' cpuset='25'/> > <vcpupin vcpu='5' cpuset='23'/> > <emulatorpin cpuset='18,20,22,24,26,28'/> > </cputune> > Li Ting, what were the results of your test? (and was it with 7.6?) Thanks! Amnon Testing looked good. and I am closing this bug. The QEMU 2.12 and RHEL 7.6 requiement should be documented. On 09/18/2018 03:47 AM, Ting Li wrote: > Hi Chris, > > I have finished to run a gating CI with qemu2.12 on dell50 i40e nic after modified the vsperf qemu.py, the performance are stable now. Following is the job and result link, please have a look. > job link: > https://beaker.engineering.redhat.com/jobs/2772869 > result link: > https://docs.google.com/spreadsheets/d/1-ZyB6vpCtOwPsXbHjPPn2uFaTSwXZS01mn0CJOtzTZo/edit#gid=1891949767 > > thanks, > Li Ting > The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |