Bug 664882 - vdsm produces too many iops
Summary: vdsm produces too many iops
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: vdsm22
Version: 5.5.z
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Ayal Baron
QA Contact: Daniel Paikov
URL:
Whiteboard: Storage
Depends On: 668247
Blocks: 666227 672238
TreeView+ depends on / blocked
 
Reported: 2010-12-22 00:50 UTC by Vladik Romanovsky
Modified: 2018-11-14 17:28 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Hosts that held the Storage Pool Manager (SPM) role performed unnecessary I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform unnecessary I/O operations when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.
Clone Of:
: 666227 (view as bug list)
Environment:
Last Closed: 2012-02-21 04:52:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0169 0 normal SHIPPED_LIVE vdsm bug fix update 2012-02-21 09:51:13 UTC

Description Vladik Romanovsky 2010-12-22 00:50:43 UTC
Hi,

SPM hosts produce ~ 2000 iops, some times even more, every 2 seconds.

This is probably because the periodic inbox reads are being done as the following:

dd if=[path_to_inbox] iflag=direct bs=512 count=2000, while the total size of it  is only 1M

This situation heavily impacts customers SAN.

Thanks,

Vladik

Comment 6 Omri Hochman 2011-01-26 09:42:30 UTC
verified with vdsm22-4.5-63.13.el5_6,


Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz
  await  svctm  %util
sda               0.01    45.52  0.03 20.20     0.68   525.75    26.02     0.20
   9.81   5.42  10.97
sda1              0.00     0.00  0.00  0.00     0.00     0.00    13.33     0.00
   4.26   4.09   0.00
sda2              0.01    45.52  0.03 20.20     0.68   525.75    26.03     0.20
   9.81   5.42  10.97
dm-0              0.00     0.00  0.04 65.72     0.68   525.75     8.01     1.20
  18.22   1.67  10.97
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     8.00     0.00
   5.14   1.31   0.00
dm-14             0.00     0.00  1.00  0.00     8.03     0.00     8.00     0.00
   0.02   0.02   0.00
dm-34             0.00     0.00  0.99  0.01     7.93     0.94     8.85     0.00
   3.11   0.02   0.00
dm-15             0.00     0.00  0.44  0.00     3.44     0.00     7.75     0.00

Comment 7 Omri Hochman 2011-01-26 10:25:59 UTC
should be re-verified for 5.7

Comment 10 Stephen Gordon 2012-02-10 21:13:53 UTC
Please provide a technical note for this bugzilla in the "Technical Notes" field, or an indication that it is not to be documented in the errata text.

Comment 11 Dan Kenigsberg 2012-02-12 15:46:47 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
C: On Red Hat Enterprise Linux 5.6 hosts that held the SPM role performed additional I/O operations when periodically querying their inbox.
C: The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage.
F: As of the vdsm22-4.5-63.14.el5_6 package vdsm does not perform additional un-necessary I/O when it periodically polls the inbox.
R: The SPM uses less I/O operations.

Text: Hosts that held the SPM role performed additional I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform additional un-necessary I/O when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.

Comment 12 Stephen Gordon 2012-02-13 15:31:34 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,6 +1 @@
-C: On Red Hat Enterprise Linux 5.6 hosts that held the SPM role performed additional I/O operations when periodically querying their inbox.
+Hosts that held the SPM role performed unnecessary I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform unnecessary I/O operations when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.-C: The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage.
-F: As of the vdsm22-4.5-63.14.el5_6 package vdsm does not perform additional un-necessary I/O when it periodically polls the inbox.
-R: The SPM uses less I/O operations.
-
-Text: Hosts that held the SPM role performed additional I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform additional un-necessary I/O when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.

Comment 13 Stephen Gordon 2012-02-13 15:34:47 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Hosts that held the SPM role performed unnecessary I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform unnecessary I/O operations when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.+Hosts that held the Storage Pool Manager (SPM) role performed unnecessary I/O operations when periodically querying their inbox. The additional I/O operations resulted in performance degradation in environments that were backed by SAN storage. With this update, VDSM does not perform unnecessary I/O operations when it periodically polls the inbox. As a result, I/O overhead on the SPM is reduced.

Comment 14 errata-xmlrpc 2012-02-21 04:52:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0169.html


Note You need to log in before you can comment on or make changes to this bug.