Bug 610344 - Excessive lseek() causes severe performance issues with vm disk images over NFS
Excessive lseek() causes severe performance issues with vm disk images over NFS
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm (Show other bugs)
5.5
All Linux
high Severity high
: rc
: ---
Assigned To: Virtualization Maintenance
Virtualization Bugs
: ZStream
Depends On: 600375
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-02 02:27 EDT by RHEL Product and Program Management
Modified: 2013-01-09 17:48 EST (History)
12 users (show)

See Also:
Fixed In Version: kvm-83-164.el5_5.14
Doc Type: Bug Fix
Doc Text:
When using the Network File System (NFS), lseek(SEEK_END) operations resulted in a GETATTR command being sent to th eserver, with the result that performance was reduced on disk images over NFS. With this update, the pread() and pwrite() functions are used instead of lseek(), read() and write(), with the result that performance is increased when using NFS.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-29 01:49:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0536 normal SHIPPED_LIVE kvm bug fix update 2010-07-29 01:49:39 EDT

  None (edit)
Description RHEL Product and Program Management 2010-07-02 02:27:17 EDT
This bug has been copied from bug #600375 and has been proposed
to be backported to 5.5 z-stream (EUS).
Comment 6 Jes Sorensen 2010-07-12 04:56:06 EDT
I doubt you are going to see much performance increase as long as you
test against qcow2 images. Also your NFS server has to be fast enough.

However to notice that the amount of system calls is down by a lot in
your sar case, so it does look like the patch is having an effect.

Cheers,
Jes
Comment 7 juzhang 2010-07-12 06:28:21 EDT
Tested again,steps is same as comment5. the only difference is using raw img instead of qcow2 img.

vmstat data:
=======

kvm-83-164.el5_5.13(Unfixed version):
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
1  0    0 5197172  27220 372132    0    0    10     1  434  104  0  2 97  0  0
2  0    0 5197048  27228 372132    0    0     0     9 11517 44952  2 38 59  0  0
2  0    0 5197048  27228 372132    0    0     0     0 11331 44827  2 40 58  0  0
2  0    0 5197048  27228 372132    0    0     0     0 11496 43912  2 38 59  0  0
1  0    0 5197048  27228 372132    0    0     0     0 9772 37414  2 32 66  0  0
1  0    0 5197172  27228 372132    0    0     0    58 9959 35621  2 34 65  0  0
3  0    0 5197048  27232 372132    0    0     0     1 11628 42410  2 39 59  0  0
3  0    0 5197048  27232 372132    0    0     0     0 11624 45053  2 37 60  0  0
0  0    0 5197048  27232 372132    0    0     0     0 4907 17974  1 15 84  0  0
0  0    0 5197072  27232 372132    0    0     0     0 1011 1942  0  0 100  0  0
1  0    0 5197112  27232 372132    0    0     0     0 1068 2489  0  1 99  0  0
1  0    0 5197204  27240 372132    0    0     0     3 11673 42265  2 38 60  0  0
2  0    0 5197196  27240 372132    0    0     0     0 11647 44516  2 37 60  0  0
0  0    0 5197072  27240 372132    0    0     0     0 5698 22621  1 19 80  0  0
0  0    0 5197072  27240 372132    0    0     0     0 1027 5136  0  2 98  0  0
3  0    0 5197196  27248 372132    0    0     0     2 8698 31932  2 28 70  0  0
1  0    0 5197196  27248 372132    0    0     0     0 6667 25798  1 22 77  0  0
0  0    0 5197196  27248 372132    0    0     0     2 3248 12823  1  9 91  0  0
1  0    0 5197320  27248 372132    0    0     0     0 5627 18967  1 31 68  0  0
0  0    0 5197180  27248 372132    0    0     0     0 4965 17001  1 31 68  0  0
2  0    0 5197320  27248 372132    0    0     0     0 1579 6378  0  3 96  0  0
2  0    0 5197444  27256 372132    0    0     0     2 2248 5584  0 27 73  0  0


kvm-83-164.el5_5.14(fixed version):

procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
3  0   0 5087828  27024 385156    0    0    71     8  795 4605  1  8 90  1  0
4  0   0 5087456  27032 385156    0    0     0     9 17904 135735  5 45 50  0  0
3  0   0 5087332  27032 385156    0    0     0     0 17936 134865  5 44 51  0  0
5  0   0 5087464  27032 385156    0    0     0     0 8702 64632  2 21 77  0  0
3  0   0 5087464  27040 385156    0    0     0     3 13229 102690  4 34 62  0  0
2  0   0 5087464  27040 385156    0    0     0     0 15709 118306  5 41 55  0  0
5  0   0 5087464  27040 385156    0    0     0     0 15576 121525  4 39 57  0  0
1  0   0 5087588  27040 385156    0    0     0     0 13034 106829  3 31 65  0  0
1  0   0 5087216  27048 385148    0    0     1     9 16383 129393  5 43 52  0  0
5  0   0 5087224  27060 385144    0    0     0    30 16087 127243  5 42 53  0  0
4  0   0 5087232  27060 385156    0    0     0     0 16650 136819  5 44 51  0  0
2  0   0 5087360  27060 385156    0    0     0     0 16733 136410  5 45 50  0  0
2  0   0 5087236  27060 385156    0    0     0     0 14545 118036  4 38 58  0  0
1  0   0 5087112  27060 385156    0    0     0     0 17316 133932  5 44 52  0  0
4  0   0 5087360  27060 385156    0    0     0     0 12300 97354  4 32 64  0  0
4  0   0 5087116  27060 385156    0    0     0     0 16594 134707  5 43 52  0  0
3  0   0 5087240  27068 385156    0    0     0    18 13164 107132  4 35 61  0  0
2  0   0 5087240  27068 385156    0    0     0     0 10655 83284  3 26 71  0  0
1  0   0 5087240  27068 385156    0    0     0     0 15966 118977  5 40 55  0  0



SAR data:
=======

kvm-83-164.el5_5.13(Unfixed version):

# sar -n NFS 5 20
Linux 2.6.18-194.3.1.el5 (dhcp-65-158.nay.redhat.com) 	07/12/2010

05:47:13 PM    call/s retrans/s    read/s   write/s  access/s  getatt/s
05:47:18 PM   3010.00      0.00      0.20   1504.80      0.00   1505.00
05:47:23 PM   3194.61      0.00      0.00   1597.41      0.00   1597.21
05:47:28 PM   3098.00      0.00      0.00   1549.00      0.00   1549.00
05:47:33 PM   2495.40      0.00      0.00   1247.60      0.00   1247.80
05:47:38 PM   2560.00      0.00      0.00   1280.00      0.00   1280.00
05:47:43 PM   3106.20      0.00      0.00   1553.20      0.00   1553.00
05:47:48 PM   3017.80      0.00      0.00   1508.80      0.00   1509.00
05:47:53 PM   2865.33      0.00      0.00   1432.67      0.00   1432.67
05:47:58 PM      0.00      0.00      0.00      0.00      0.00      0.00
05:48:03 PM      0.00      0.00      0.00      0.00      0.00      0.00
05:48:08 PM   1598.40      0.00      1.40    797.80      0.00    799.20
05:48:13 PM   3121.16      0.00      0.00   1560.68      0.00   1560.48
05:48:18 PM   3007.60      0.00      0.00   1503.80      0.00   1503.80
05:48:23 PM      0.00      0.00      0.00      0.00      0.00      0.00
05:48:28 PM    626.51      0.00      0.00    313.25      0.00    313.25
05:48:33 PM   3151.90      0.00      0.00   1575.85      0.00   1576.05
05:48:38 PM    148.90      0.00      0.00     74.55      0.00     74.35
05:48:43 PM   1981.44      0.00      0.00    990.62      0.00    990.82
05:48:48 PM      0.00      0.00      0.00      0.00      0.00      0.00
05:48:53 PM   1218.60      0.00      0.00    609.40      0.00    609.20
Average:      1910.62      0.00      0.08    955.23      0.00    955.30

kvm-83-164.el5_5.14(Unfixed version):

[root@dhcp-65-158 ~]# sar -n NFS 5 20
Linux 2.6.18-194.8.1.el5 (dhcp-65-158.nay.redhat.com) 	07/12/2010

06:17:47 PM    call/s retrans/s    read/s   write/s  access/s  getatt/s
06:17:52 PM   1838.80      0.00      0.00   1838.80      0.00      0.00
06:17:57 PM   1834.40      0.00      0.60   1833.80      0.00      0.00
06:18:02 PM   1213.57      0.00      0.20   1213.37      0.00      0.00
06:18:07 PM    941.08      0.00      3.01    938.08      0.00      0.00
06:18:12 PM   1621.16      0.00      2.59   1618.56      0.00      0.00
06:18:17 PM   1595.19      0.00      0.00   1595.19      0.00      0.00
06:18:22 PM   1312.57      0.00      0.00   1312.57      0.00      0.00
06:18:27 PM   1657.92      0.00      0.00   1657.92      0.00      0.00
06:18:32 PM   1643.11      0.00      0.80   1642.32      0.00      0.00
06:18:37 PM   1704.20      0.00      1.40   1702.80      0.00      0.00
06:18:42 PM   1737.00      0.00      0.00   1737.00      0.00      0.00
06:18:47 PM   1478.20      0.00      0.00   1478.20      0.00      0.00
06:18:52 PM   1783.37      0.00      0.20   1783.17      0.00      0.00
06:18:57 PM   1657.49      0.00      0.00   1657.49      0.00      0.00
06:19:02 PM   1300.40      0.00      0.00   1300.40      0.00      0.00
06:19:07 PM   1628.00      0.00      0.60   1627.40      0.00      0.00
06:19:12 PM   1009.58      0.00      0.00   1009.58      0.00      0.00
06:19:17 PM   1512.60      0.00      0.80   1511.80      0.00      0.00
06:19:22 PM    828.26      0.00      0.20    828.06      0.00      0.00
06:19:27 PM    754.60      0.00      0.00    754.60      0.00      0.00
Average:      1452.61      0.00      0.52   1452.09      0.00      0.00


Jes,please review the results.
Comment 8 Jes Sorensen 2010-07-12 08:58:59 EDT
Hi,

Thanks for the data. It is showing exactly what we expect to see.
If we look at the fixed version kvm-83-164.el5_5.14 we see a write/s
rate that pretty much matches the call/s rate, where as in the
unfixed version we see the call/s rate being almost double the write/s
rate, ie. it issues both an lseek() and a write() for each transaction.

Looks good!

Cheers,
Jes
Comment 12 Eduardo Habkost 2010-07-23 14:55:49 EDT
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
Cause: Excessive lseek() operations being performed in KVM's I/O code path.  When
using NFS, the lseek(SEEK_END) operations involve sending a GETATTR command to
the server.

Consequence: poor performance on disk images over NFS>

Fix: Use pread/pwrite instead of lseek+read/write

Result: better perfomance on disk images over NFS
Comment 13 Douglas Silas 2010-07-28 11:41:32 EDT
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,9 +1 @@
-Cause: Excessive lseek() operations being performed in KVM's I/O code path.  When
+When using the Network File System (NFS), lseek(SEEK_END) operations resulted in a GETATTR command being sent to th eserver, with the result that performance was reduced on disk images over NFS. With this update, the pread() and pwrite() functions are used instead of lseek(), read() and write(), with the result that performance is increased when using NFS.-using NFS, the lseek(SEEK_END) operations involve sending a GETATTR command to
-the server.
-
-Consequence: poor performance on disk images over NFS>
-
-Fix: Use pread/pwrite instead of lseek+read/write
-
-Result: better perfomance on disk images over NFS
Comment 14 errata-xmlrpc 2010-07-29 01:49:46 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0536.html

Note You need to log in before you can comment on or make changes to this bug.