RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1359510 - RFE: Request for warning if multipathd is not running/active
Summary: RFE: Request for warning if multipathd is not running/active
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: device-mapper-multipath
Version: 7.2
Hardware: x86_64
OS: Linux
medium
low
Target Milestone: rc
: ---
Assignee: Ben Marzinski
QA Contact: Lin Li
Steven J. Levine
URL:
Whiteboard:
Depends On: 1305589
Blocks: 1298243 1385242
TreeView+ depends on / blocked
 
Reported: 2016-07-24 13:21 UTC by Milan P. Gandhi
Modified: 2021-09-03 12:04 UTC (History)
16 users (show)

Fixed In Version: device-mapper-multipath-0.4.9-101.el7
Doc Type: Enhancement
Doc Text:
Warning messages when `multipathd` is not running Users now get warning messages if they run a `multipath` command that creates or lists multipath devices while `multipathd` is not running. If `multipathd` is not running, then the devices are not able to restore paths that have failed or react to changes in the device setup. The `multipathd` daemon now prints a warning message if there are multipath devices and `multipathd` is not running.
Clone Of: 1305589
Environment:
Last Closed: 2017-08-01 16:34:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Proposed Patch v1 (3.11 KB, patch)
2016-07-24 13:28 UTC, Milan P. Gandhi 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2139121 0 None None None 2016-07-24 13:21:52 UTC
Red Hat Product Errata RHBA-2017:1961 0 normal SHIPPED_LIVE device-mapper-multipath bug fix and enhancement update 2017-08-01 17:56:09 UTC

Description Milan P. Gandhi 2016-07-24 13:21:53 UTC 
+++ This bug was initially created as a clone of Bug #1305589 +++

Description of problem:

Opening at customer request.  Is it possible to get a warning if the dm_multipath module is loaded, multipath devices are present, but multipathd is not set to on?  Perhaps when using the 'multipath' command.  The idea is that most users never use the multipathd command; they configure multipathing and use the multipath command. 

The multipathd service is an integral part of multipathing, so we should at least give an informative message if it is not running and the system is using multipath.

Why do we not auto-start multipathd if the module is in use?  Are there use cases where we would want to have multipath devices, but not use multipathd?  I have seen many cases where proper failover/mapping was not acheived because multipathd was not running and no one knew.  Could we request an auto load for multipathd instead of the warning?  Maybe both?  

Thanks for your time.  Please let me know if I can be of any help.

Version-Release number of selected component (if applicable):

device-mapper-multipath-0.4.9-87.el6.x86_64

Expected Results:

- multipathd service autostart with module load
- warning if service is not running when using multipath command

--- Additional comment from Ben Marzinski on 2016-02-08 15:04:02 EST ---

Loading the module can't autostart the service. There's no hook for anything like that in the module code. Besides, if there are no multipath devices, we don't want to start the service.  If the service is enabled in systemd, it will autostart on boot, as long a /etc/multipath.conf exists.  Creating the configuration file the recommended way, with

# mpathconf --enable

does enable the service in systemd. However, it does not autostart the service at that point.  This is because users often want to actually edit the configuration file before starting the service the first time. The service will autostart on the next reboot.

There are two ways to actually create multipath devices. Either multipathd will automatically create them, or they can be created manually by running multipath.  I can make multipath print a warning message if multipathd isn't running. This should solve the case where there are multipath devices but multipathd isn't running.

Does this sound reasonable?

--- Additional comment from John Pittman on 2016-02-08 15:12:22 EST ---

Yes Ben that sounds perfect.
Thanks a lot.

John

--- Additional comment from Milan P. Gandhi on 2016-07-24 09:19 EDT ---

With the attached patch, dm-multipath commands e.g. multipath -v2, multipath -ll etc. now checks if there are multipath device maps created, and multipathd service is running or not?

If the multipath device maps are created, but multipathd service is not running, then in such case there will be a warning message displayed to inform the user that IO failover/failback may not work as expected without multipathd process running.
Comment 1 Milan P. Gandhi 2016-07-24 13:28:12 UTC 
Created attachment 1183370 [details]
Proposed Patch v1
Comment 2 Milan P. Gandhi 2016-07-24 13:35:42 UTC 
dm-multipath command outputs before applying the attached patch:


[root@testhost ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@testhost ~]# uname -a
Linux testhost 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@testhost ~]# rpm -qa|grep -i multipath
device-mapper-multipath-0.4.9-85.el7_2.5.x86_64
device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64
[root@testhost ~]# 


Currently "multipathd" service is running: 


[root@testhost ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-07-24 17:06:31 IST; 4min 4s ago
 Main PID: 517 (multipathd)
   CGroup: /system.slice/multipathd.service
           └─517 /sbin/multipathd
[...]
[root@testhost ~]# 
[root@testhost ~]# 


Stop the "multipathd" service:


[root@testhost ~]# systemctl stop multipathd
[root@testhost ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2016-07-24 17:10:42 IST; 2s ago
 Main PID: 517 (code=exited, status=0/SUCCESS)

[...]
Jul 24 17:10:42 testhost systemd[1]: Stopped Device-Mapper Multipath Device Controller.
[root@testhost ~]# 


With "multipathd" service stopped, we do not get any warnings for "multipath -v2, -ll, -v4" commands:


[root@testhost ~]# multipath -ll
mpathd (1IET_00050004) dm-5 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:4 sdd 8:48 active ready running
mpathc (1IET_00050003) dm-4 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:3 sdc 8:32 active ready running
mpathb (1IET_00050002) dm-3 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:2 sdb 8:16 active ready running
mpatha (1IET_00050001) dm-2 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:1 sda 8:0  active ready running
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# multipath -v2
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# multipath -F
[root@testhost ~]# multipath -v2
create: mpatha (1IET_00050001) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:1 sda 8:0  undef ready running
create: mpathb (1IET_00050002) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:2 sdb 8:16 undef ready running
create: mpathc (1IET_00050003) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:3 sdc 8:32 undef ready running
create: mpathd (1IET_00050004) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:4 sdd 8:48 undef ready running
[root@testhost ~]# 
[root@testhost ~]#
Comment 4 Milan P. Gandhi 2016-07-24 13:40:17 UTC 
Using the test dm-multipath packages containing the attached patch:


Currently "multipathd" service is running on the server:
=======================================================

[root@testhost ~]# 
[root@testhost ~]# ps aux|grep -i multip
root       520  0.0  0.6 252732  6920 ?        SLl  18:13   0:00 /sbin/multipathd
root      2464  0.0  0.0 112652   980 pts/0    S+   18:16   0:00 grep --color=auto -i multip
[root@testhost ~]# 


Now stop the "multipathd" service:
=================================

[root@testhost ~]# systemctl stop multipathd
[root@testhost ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2016-07-24 18:17:52 IST; 4s ago
  Process: 516 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 507 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 488 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 520 (code=exited, status=0/SUCCESS)

[...]
Jul 24 18:17:52 testhost systemd[1]: Stopped Device-Mapper Multipath Device Controller.
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# ps aux|grep -i multip
root      2613  0.0  0.0 112648   980 pts/0    R+   18:18   0:00 grep --color=auto -i multip
[root@testhost ~]# 
[root@testhost ~]# 


The "multipath -v2", "multipath -ll" commands now verifies if there is at least
one multipath device present on server, if yes, it will check if "multipathd"
process is running or not.

Since there are multipath device maps present, but "multipathd" service is not
running, we get the warning as seen in following command output:
=======================================================


[root@testhost ~]# multipath -ll
mpathd (1IET_00050004) dm-5 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:4 sdd 8:48 active ready running
mpathc (1IET_00050003) dm-4 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:3 sdc 8:32 active ready running
mpathb (1IET_00050002) dm-3 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:2 sdb 8:16 active ready running
mpatha (1IET_00050001) dm-2 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:1 sda 8:0  active ready running
Jul 24 18:18:08 | multipath device maps are present, but 'multipathd' service is not running
Jul 24 18:18:08 | IO failover/failback will not work without 'multipathd' service running
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# multipath -v2
Jul 24 18:18:12 | multipath device maps are present, but 'multipathd' service is not running
Jul 24 18:18:12 | IO failover/failback will not work without 'multipathd' service running
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# multipath -ll
mpathd (1IET_00050004) dm-5 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:4 sdd 8:48 active ready running
mpathc (1IET_00050003) dm-4 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:3 sdc 8:32 active ready running
mpathb (1IET_00050002) dm-3 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:2 sdb 8:16 active ready running
mpatha (1IET_00050001) dm-2 IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:1 sda 8:0  active ready running
Jul 24 18:18:16 | multipath device maps are present, but 'multipathd' service is not running
Jul 24 18:18:16 | IO failover/failback will not work without 'multipathd' service running
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# 
[root@testhost ~]# multipath -F
[root@testhost ~]# 
[root@testhost ~]# multipath -v2
create: mpatha (1IET_00050001) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:1 sda 8:0  undef ready running
create: mpathb (1IET_00050002) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:2 sdb 8:16 undef ready running
create: mpathc (1IET_00050003) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:3 sdc 8:32 undef ready running
create: mpathd (1IET_00050004) undef IET     ,VIRTUAL-DISK    
size=512M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
  `- 3:0:0:4 sdd 8:48 undef ready running
Jul 24 18:18:23 | 'multipathd' service is currently not running, IO failover/failback will not work
[root@testhost ~]#
Comment 6 Mark Thacker 2016-11-23 16:14:06 UTC 
Adding pm_ack for this.
Comment 7 Ben Marzinski 2017-02-16 18:32:32 UTC 
Patch applied. Thanks.
Comment 10 Steven J. Levine 2017-05-08 18:05:02 UTC 
Ben:  Could you check my edit to the doc text description for this feature (just a slight rearrangement for the release note format).

Steven
Comment 11 Ben Marzinski 2017-05-08 18:40:06 UTC 
Looks fine.
Comment 12 Steven J. Levine 2017-05-16 16:23:10 UTC 
Adding a "topic" summary sentence to the doc text.  I had thought the first paragraph would serve as a fine summary, but it turns out to be too many characters.  I didn't change anything in the approved description.
Comment 13 Lin Li 2017-05-22 11:05:13 UTC 
Verified on device-mapper-multipath-0.4.9-111
1. [root@storageqe-84 ~]# rpm -qa | grep multipath
device-mapper-multipath-debuginfo-0.4.9-111.el7.x86_64
device-mapper-multipath-libs-0.4.9-111.el7.x86_64
device-mapper-multipath-0.4.9-111.el7.x86_64
device-mapper-multipath-sysvinit-0.4.9-111.el7.x86_64
device-mapper-multipath-devel-0.4.9-111.el7.x86_64

2. [root@storageqe-84 ~]# mpathconf --enable

3. [root@storageqe-84 ~]# service multipathd restart
Restarting multipathd (via systemctl):  [  OK  ]

4. [root@storageqe-84 ~]# multipath -ll
mpathb (353333330000007d0) dm-5 Linux   ,scsi_debug      
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 6:0:0:0 sdd 8:48 active ready running
mpatha (360fff19abdd9552f8a36e5355226ba27) dm-0 EQLOGIC ,100E-00         
size=50G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 4:0:1:0 sdb 8:16 active ready running

5. [root@storageqe-84 ~]# service multipathd stop
Stopping multipathd (via systemctl):  [  OK  ]

6. [root@storageqe-84 ~]# multipath -ll
mpathb (353333330000007d0) dm-5 Linux   ,scsi_debug      
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 6:0:0:0 sdd 8:48 active ready running
mpatha (360fff19abdd9552f8a36e5355226ba27) dm-0 EQLOGIC ,100E-00         
size=50G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 4:0:1:0 sdb 8:16 active ready running
May 22 13:00:34 | multipath device maps are present, but 'multipathd' service is not running            ------------------------->
May 22 13:00:34 | IO failover/failback will not work without 'multipathd' service running           ------------------------>
----------------->List multipath devices while multipathd is stopped, multipath  prints a warning message now.
Comment 14 errata-xmlrpc 2017-08-01 16:34:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1961
Note You need to log in before you can comment on or make changes to this bug.