RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 814617 - NFS performance regression in large file sequential writes.
Summary: NFS performance regression in large file sequential writes.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Paolo Bonzini
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-04-20 09:27 UTC by Quan Wenli
Modified: 2013-01-10 00:54 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.287.el6
Doc Type: Bug Fix
Doc Text:
No Documentation Needed
Clone Of:
Environment:
Last Closed: 2012-06-20 11:47:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
ffsb large_file_creates_256k_1.profile (928 bytes, text/plain)
2012-04-20 11:33 UTC, Quan Wenli
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0746 0 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2012-06-19 19:31:48 UTC

Description Quan Wenli 2012-04-20 09:27:13 UTC
Description of problem:

Do kvm block regresson test w/ (1Gb link) nfs backend between rhel6.2GA and rhel6.3 beta1 hosts .there is 35% performance regression in large file sequential writes.

Detailed result:
================================================================
raw format in LargeFile Creates(256KB)
    
  w/kernel-257 guest       threads|       IOPS|   Thro(MBps)|  
rhel6.2GA                        1|     195.35|         48.8|  
r63 kernel-267 qemu-272          1|     126.29|         31.6|   
diff %                            |      -35.4|        -35.2|   

qcow2 format in LargeFile Creates(256KB)
  w/kernel-257 guest       threads|       IOPS|   Thro(MBps)|  
rhel6.2GA                        1|     199.77|         49.9|
r63 kernel-267 qemu-272          1|     130.61|         32.7|
diff %                            |      -34.6|        -34.5| 

================================================================

Version-Release number of selected component (if applicable):

(host)kernel-2.6_32-262
qemu-kvm-0.12.1.2-257
(guest)kernel-2.6_32-257

How reproducible:

always 

Steps to Reproduce:
1.mount 1gb netapp server in client. 
cat /proc/mounts

192.168.0.113:/vol/s2wquan116171nfs /home/kvm_autotest_root/images nfs rw,sync,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.113,mountvers=3,mountport=4046,mountproto=tcp,local_lock=none,addr=192.168.0.113 0 0

3. create 20G raw/qcow2 images.
qemu-img create -f raw /home/kvm_autotest_root/images/storage2.raw 20G
qemu-img create -f qcow2 /home/kvm_autotest_root/images/storage2.qcow2 110G -o preallocation=metadata

4.Pass this raw/qcow2 file image to the KVM guest as a block device and use
cache=none aio=threads for the KVM guest.
/usr/libexec/qemu-kvm -name vm1 -nodefaults -vga std -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20120418-021037-GUK9,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20120418-021037-GUK9,path=/tmp/serial-20120418-021037-GUK9,server,nowait -device isa-serial,chardev=serial_id_20120418-021037-GUK9 -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 -drive file=/usr/local/autotest/tests/kvm/images/RHEL-Server-6.3-64-virtio.raw,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,readonly=off,format=raw,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/usr/local/autotest/tests/kvm/images/test.raw,index=2,if=none,id=drive-virtio-disk2,media=disk,cache=none,boot=off,snapshot=off,readonly=off,format=raw,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -device virtio-net-pci,netdev=idExu1N2,mac=9a:ff:15:58:09:ce,id=ndev00idExu1N2,bus=pci.0,addr=0x3 -netdev tap,id=idExu1N2,vhost=on  -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu Westmere -device usb-tablet,id=usb-tablet1,bus=usb1.0 -vnc :1 -vga qxl -rtc base=utc,clock=host,driftfix=slew -M rhel6.3.0 -boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -enable-kvm

4)In the KVM guest (running RHEL6.3-257), write filed in second disk (dev/vdb). ,then format the virtual disk with ext4 and mount the virtual disk in the guest.
  mkfs.ext4 /dev/vdb; mount /dev/vdb /mnt

5)run ffsb test.define direct io is on in configuration of ffsb test
# ffsb large_file_creates_256k_1.ffsb

Actual results:


Expected results:


Additional info:
this bug may caused by linearized QEMU hack removal patch in bug #767606.

Comment 2 Quan Wenli 2012-04-20 11:33:57 UTC
Created attachment 578965 [details]
ffsb large_file_creates_256k_1.profile

Comment 3 Dor Laor 2012-04-22 08:21:17 UTC
Hi Wenli, thanks for the report.
Can you please test the qemu version prior to bug #767606 fix?
We'll need to decide which component/release introduced the regression. Please help us do the bisect;

Why didn't you move #767606 back into assigned state if you're sure that's the source cause as noted in https://bugzilla.redhat.com/show_bug.cgi?id=767606#c11

Comment 4 Quan Wenli 2012-04-23 04:53:34 UTC
(In reply to comment #3)
> Hi Wenli, thanks for the report.
> Can you please test the qemu version prior to bug #767606 fix?
> We'll need to decide which component/release introduced the regression. Please
> help us do the bisect;
Since qemu version-237 which is prior to bug #767606 fix was deleted in brew,I rebuild qemu-kvm-237 packages in https://brewweb.devel.redhat.com/taskinfo?taskID=4326145.

Do same test w/ switching qemu-kvm back to 237(keep same host/guest kernel), get about 49Mb/s in 256kb sequential writes on raw format, it is quite same with rhel6.2 result in comment #0.

> 
> Why didn't you move #767606 back into assigned state if you're sure that's the
> source cause as noted in https://bugzilla.redhat.com/show_bug.cgi?id=767606#c11
yes, I should move it back to assigned immediately rather then waiting for customer's reply.

Comment 6 Ademar Reis 2012-04-23 22:21:09 UTC
From Paolo's last update on bug 767606 (c#16): "Kernel bug reopened as bug 815265, moved QE tracker to bug 814617, using that one to post the revert."

Comment 9 Quan Wenli 2012-05-02 06:47:07 UTC
Verified it w 2.6.32-269.el6.x86_64 and qemu-kvm-0.12.1.2-2.287.el6 by using  same steps in comment #0.
The result of FFSB on large file sequential writes(256k) is 46.9 Mb/s , there is no regression compared with rhel6.2host.

Base on it,change it to verified.

Comment 10 Michal Novotny 2012-05-04 13:32:28 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No Documentation Needed

Comment 11 errata-xmlrpc 2012-06-20 11:47:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0746.html


Note You need to log in before you can comment on or make changes to this bug.