Bug 484883 - provide easy way see performance metrics of disk paths in multipath tools
provide easy way see performance metrics of disk paths in multipath tools
Status: ASSIGNED
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
7.0
All Linux
low Severity medium
: pre-dev-freeze
: ---
Assigned To: Bryn M. Reeves
Lin Li
: FutureFeature
Depends On:
Blocks: 697866 756082 846704
  Show dependency treegraph
 
Reported: 2009-02-10 09:38 EST by david.schramm
Modified: 2017-02-04 00:46 EST (History)
23 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description david.schramm 2009-02-10 09:38:46 EST
Description of problem: Typical SAN connection with dual HBA's and 4 paths to a LUN. There is no way to easily see the IO distribution down each path, or if there is, it is not readily available information. Using iostat and manually piecing the parts of the puzzle together is a real time sink. Just make some option like "multipath -io" or something which would then return the mpath device, all the specific paths, and the instantaneous IO down each path including options to see errors, dropped packets, etc. The most important thing though is probably IOPS and queued IOs. In our shop using all CLARiiONs, every host has dual HBA and for each HBA at least 2 targets for every disk. So, if a host has 10 LUNS, that's 40 sd's so try and sort with iostat and still no way to see queued IOs.

Version-Release number of selected component (if applicable): 
multipath-tools v0.4.7 (03/12, 2006)

How reproducible:
Always


Steps to Reproduce: Not possible unless you 
1. run multipath -ll
2. take output of that and then run iostat -x
3. try to look at just the specific paths for the mpath device in question..good luck.
  
Actual results: a large matrix of sd? and dm? devices which mean nothing to you if you can't easily relate them back to your /dev/mapper/mpath#'s.


Expected results:
Something like this as seen with emc tool powrpath:

CLARiiON ID=APM00052300134 [dbs003]
Logical device ID=50060160B06025BF50060160B06025BF
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=Unknown, current=Unknown Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
2305 pci@9,600000/lpfc@1       c2t0d1s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d21s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d22s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d23s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d24s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d25s0 SP B0     active  dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d26s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d28s0 SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d2s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d3s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d4s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d5s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t0d7s0  SP B0     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d1s0  SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d21s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d22s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d23s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d24s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d25s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d26s0 SP A1     active  dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d28s0 SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d2s0  SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d3s0  SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d4s0  SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d5s0  SP A1     unlic   dead       0      0
2305 pci@9,600000/lpfc@1       c2t1d7s0  SP A1     unlic   dead       0      0

Pseudo name=emcpower6a
CLARiiON ID=APM00052300134 [dbs003]
Logical device ID=600601601AB01400341FB018FFF9D911 [LUN 27 >dbs003 Cobra (DB) dev 3]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
2305 pci@9,600000/lpfc@1       c2t0d27s0 SP B0     active  alive      0      2
2305 pci@9,600000/lpfc@1       c2t1d27s0 SP A1     active  alive      0      0

Pseudo name=emcpower14a
CLARiiON ID=APM00052300134 [dbs003]
Logical device ID=600601601AB01400EAFEB491F346DA11 [LUN 39 >dbs003 Cobra (ARCH) dev 3]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
2305 pci@9,600000/lpfc@1       c2t0d6s0  SP B0     active  alive      0      2
2305 pci@9,600000/lpfc@1       c2t1d6s0  SP A1     active  alive      0      0



Additional info: Or, the other cool thing would be to provide IO data similar to how Veritas does it with DMP....this is very helpful when doing load testing or troubleshooting performance problems...

[machine:man]: vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
c9              EMC_CLARiiON    ENABLED      EMC_CLARiiON0
c8              EMC_CLARiiON    ENABLED      EMC_CLARiiON0
c1              Disk            ENABLED      Disk
[machine:man]: vxdmpadm getsubpaths ctlr=c8
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-TYPE   ENCLR-NAME   ATTRS
================================================================================
c8t30d0s2    ENABLED    SECONDARY    c7t32d0s2    EMC_CLARiiON EMC_CLARiiON0   -
c8t30d1s2    ENABLED(A) PRIMARY      c7t32d1s2    EMC_CLARiiON EMC_CLARiiON0   -
c8t33d0s2    ENABLED(A) PRIMARY      c7t32d0s2    EMC_CLARiiON EMC_CLARiiON0   -
c8t33d1s2    ENABLED    SECONDARY    c7t32d1s2    EMC_CLARiiON EMC_CLARiiON0   -

[machine:man]: vxdmpadm getsubpaths ctlr=c9
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-TYPE   ENCLR-NAME   ATTRS
================================================================================
c9t31d0s2    ENABLED    SECONDARY    c7t32d0s2    EMC_CLARiiON EMC_CLARiiON0   -
c9t31d1s2    ENABLED(A) PRIMARY      c7t32d1s2    EMC_CLARiiON EMC_CLARiiON0   -
c9t32d0s2    ENABLED(A) PRIMARY      c7t32d0s2    EMC_CLARiiON EMC_CLARiiON0   -
c9t32d1s2    ENABLED    SECONDARY    c7t32d1s2    EMC_CLARiiON EMC_CLARiiON0   -
[machine:man]: vxdmpadm iostat show ctlr=c8
                       cpu usage = 343us    per cpu memory = 32768b
                         OPERATIONS             KBYTES            AVG TIME(ms)
PATHNAME              READS     WRITES      READS     WRITES     READS   WRITES
c8t33d1s2                 0          0          0          0  0.000000 0.000000
c8t30d1s2                 0          7          0          0  0.000000 0.000000
c8t30d0s2                 0          0          0          0  0.000000 0.000000
c8t33d0s2               434          6         17          0 196.764706 0.000000
[machine:man]: vxdmpadm iostat show ctlr=c9
                       cpu usage = 349us    per cpu memory = 32768b
                         OPERATIONS             KBYTES            AVG TIME(ms)
PATHNAME              READS     WRITES      READS     WRITES     READS   WRITES
c9t31d1s2                 0          9          0          0  0.000000 0.000000
c9t32d1s2                 0          0          0          0  0.000000 0.000000
c9t31d0s2                 0          0          0          0  0.000000 0.000000
c9t32d0s2               418          7         16          0 290.125000 0.000000
Comment 8 Tom Lavigne 2012-09-07 11:18:09 EDT
This request was evaluated by Red Hat Product Management for 
inclusion in the current release of Red Hat Enterprise Linux.
Since we are unable to provide this feature at this time,  
it has been proposed for the next release of 
Red Hat Enterprise Linux.
Comment 9 Alasdair Kergon 2013-10-09 07:13:02 EDT
(We're also working on some new code for better I/O statistics for all device-mapper devices.)
Comment 10 Bryn M. Reeves 2014-11-21 09:29:54 EST
Note that dm statistics are not available in RHEL6/7 for device-mapper multipath devices without additional manual steps being taken.

This is because the dm-statistics support is only available for bio-based targets and the dm-multipath target is request-based in both these releases.

This means that to use the extra functionality (beyond that offered by /proc/diskstats - i.e. counters for arbitrary sub-regions) requires a dm-linear layer to be stacked on top of the multipath map.

This means we can fairly easily provide dm stats for /partitions/ of multipath devices (since they already use a dm-linear layer managed by kpartx) but whole-disk statistics need an additional dm device to be created and stacked between the multipath layer and any partition mappings:

# multipath -l mpatht
mpatht (3600601608e661a00e4710bf88370e211) dm-3 DGC,RAID 5
size=5.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 4:0:0:0 sdb 8:16  active undef  running
  `- 5:0:0:0 sdh 8:112 active undef  running

# dmsetup ls --tree 
tvg0-tlv0 (253:11)
 └─mpathtp1 (253:7)
    └─mpatht (253:3)
       ├─ (8:112)
       └─ (8:16)
[..]

Trying to create stats regions on the mpath device will fail:

# dmsetup message mpatht 0 @stats_create - /1 s1
device-mapper: message ioctl on mpatht failed: Operation not supported
Command failed
# dmesg | tail -1
device-mapper: stats: Statistics are only supported for bio-based devices


The single partition mapping allows them:

# dmsetup message mpathtp1 0 @stats_create - /4 s1
2
# mke2fs -j /dev/tvg0/tlv0 &>/dev/null
# dmsetup message mpathtp1 0 @stats_print 2
0+512000 138 0 1104 3558 315 0 2520 47652 0 542 51210 325 217
512000+512000 0 0 0 0 0 0 0 0 0 0 0 0 0
1024000+512000 0 0 0 0 0 0 0 0 0 0 0 0 0
1536000+512000 0 0 0 0 0 0 0 0 0 0 0 0 0

It's also possible now to get iostat to automatically use the device-mapper names when displaying stats for dm devices (-N).

In RHEL7 sysstat supports device grouping, e.g.:

# iostat -xN mpathtp1 -g mpatht { sdb sdh }
 Linux 2.6.32-494.el6.x86_64 (pe1950-5.gsslab.fab.redhat.com) 	21/11/14 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.03    0.00    0.03    0.00    0.00   99.95

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
mpathtp1          0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00   82.11   1.04   0.00
mpatht            0.00     0.00    0.00    0.00     0.00     0.00    14.34     0.00    3.17   1.50   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00    11.24     0.00    1.70   1.28   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00    11.86     0.00    2.54   1.50   0.00

Which may provide another alternative for some users until more complete dm stats support is available. The need to specify subordinate devices on the command line is a bit awkward but this could be scripted easily if required.

Note You need to log in before you can comment on or make changes to this bug.